Automated and Intelligent Data Migration Strategy in High Energy Physical Storage Systems

As a data-intensive computing application, high-energy physics requires storage and computing for large amounts of data at the PB level. IHEP computing center is beginning to use tiered storage architectures, such as tape, disk or solid state drives to reduce hardware purchase costs and power consumption. At present, automatic data migration strategies are mainly used to resolve data migration between memory and disk. So the rules are relatively simple. This paper attempted to use the deep learning algorithm model to predict the evolution trend of data access heat as the basis for data migration. The implementation of some initial parts of the system were discussed, as well as the file trace collector and the LSTM model. At last some preliminary experiments are conducted with these parts.


Introduction
With data-intensive computing [1] requirements, high-energy physics computing needs to store and process massive physical data at the PB level every year. The data access performance of the storage system is the key factor. More and more high-energy physics experiment data puts higher and higher requirements on storage systems, including capacity, performance, scalability, reliability, long-term preservation and cost performance. On one hand, storage systems based on traditional mechanical hard disks or tapes cannot provide higher IOPS(read/write bandwidth cannot be further improved) and the unit price of new solid-state disks is still high, so it has no advantage in terms of storage capacity and cost performance. On the other hand, according to one survey [2], only a small part of the data in the storage system is active during a short period of time, while most of the data will not be accessed. The Chinese Academy of Sciences High Energy Computing Center plans to use hierarchical storage system with a unified namespace to store HEP data, including tape, traditional mechanical hard drives and solidstate drives.
To improve the resource utilization of the storage system, a migration strategy is needed to migrate data between different storage tiers. Heuristic algorithms based on personal experience of system administrators are currently used, which may have empirical biases and lack of load versatility and adaptability. Two major massive storage systems, such as Lustre [3] and EOS [4], provide hierarchical storage techniques. But data migration between all levels of storage is completely specified by the administrator through the file migration list, which has a lot of labor costs. In high-energy physics storage systems, data access requests are usually not completely random. They are closely related to the user/application computing model [5]. This paper first introduces the research status and related work basis of the data migration strategy, then it introduces the collection of file access training samples from the actual high-energy physical storage system. Finally, it presents how we trained a predictive model with supervised learning to predict future access of files based on the deep learning algorithm, and further developed migration rules based on changes in predicted access heat.

Research status
Data migration strategies have always been an important area of research for storage systems. Literature [6] reviews describe algorithm-driven file migration strategies such as LRU, CLOCK, 2Q, GDSF, LFUDA and FIFO. These strategies are mainly used to solve the problem of data migration between volatile main storage and external disks in a stand-alone computer. The essence is to set a threshold on file access feature (such as the last access time). To run in the operating system kernel, a trade-off must be made between prediction accuracy and calculation efficiency, making it impossible to exploit complex file access features or build complex models. Further literature [7] introduced the ULNS model which would consider user information for predictions. The prediction accuracy is better than the normal LS model, but it also relies on a large number of historical file access sequences. Other literature [8] proposed the idea of formulating migration rules based on prediction of file access frequency. The method assumed that each time a user would read the files in a complete and sequential manner. A supervised learning model based on the support vector machine (SVM) algorithm was trained to perform the prediction task. It obtained good prediction results on the specific WEB data set. The SVM algorithm is difficult to implement for large-scale training samples because it supports vectors by means of quadratic programming, which would consume a lot of machine memory and computation time. In recent years, technology based on deep learning broke through the limitations of traditional neural networks on the number of layers and the number of nodes per layer. The training methods are quite different from the traditional machine learning algorithms. This paper uses a deep learning algorithm based on long short-term memory neural networks to solve this problem.

Related work 3.1 EOS Massive Distributed Storage System
Distributed storage systems are an integral part of a high-energy physics computing environment. In 2011, in order to meet the data storage requirements of the LHC Run II experiment, the EOS mass distributed storage system was put into use. Currently CERN EOS manages 15 instances, with more than 2.6 billion files and nearly 250 PB of data. The IHEP computing center also uses EOS to manage more than 10PB of data, and serves more than 1000 users in LHAASO [9]. EOS is composed of management server (MGM), message queue (MQ) and file storage server (FST) based on the Xrootd [10] protocol framework. The file storage server supports a variety of storage media such as mechanical hard disks and solid-state drives. At the same time, EOS CTA services support data reading and writing in the tape library. EOS divides the file storage server into multiple storage groups. RAID mode is used for file backup and striping. EOS has a unified namespace. Data and files in different storage groups do not affect each other. Files can be migrated between groups through EOS admin commands. What's more, EOS has designed a comprehensive hierarchical logging mechanism.

LSTM(long-term and short-term memory neural network)
Long-Short Term Memory Network(LSTM) is an improved Recurrent Neural Network (RNN) [11]. The neural network was originally a machine learning method based on the human brain. However, in general, humans do not start thinking from a blank brain all the time. For example, when reading an article, human beings infer the meaning of the current text based on the knowledge already possessed Due to the problem of gradient disappearance in traditional cyclic neural networks, it is difficult to deal with long-term dependent information in learning sequences. FA Gerset al [12]. proposed the network structure of LSTM, which redesigned the computational nodes in the cyclic neural network It uses time memory cells to record the state of the current moment. Those cells are commonly referred to as longshort-term memory neural networks. Each cell has three information transfer gates: the input gate, the output gate, and the forgetting gate as shown in fig1. The information switch door can selectively pass information. The forgetting gate determines in the sequence data at a certain moment, what information is discarded from the cell, and returns the output by a value between 0 and 1 for each cell state −1 (0 means completely discarded, 1 means completely reserved) ). ℎ −1 represents the output of the previous LSTM Cell, represents the input of the current LSTM Cell, and α represents the sigmod function. = * −1 + * ˜ The output gate determines which portion of the cell state to output via a sigmod layer based on the updated LSTM cell state. And a tanh function was used to ensure that only the portion of the LSTM cell that determines the output. = ( [ℎ −1 , ] + ) ℎ = * ℎ( ) The three gates jointly control the inflow and outflow of information, as well as the update of LSTM cell status. Therefore, the LSTM model excels at mining long-term or short-term dependency information in a time series, and is suitable for predicting delayed events with time intervals.

Design and Implementation
As shown in the figure 2, the system interacts with high-energy physics storage systems such as Lustre and EOS through an intelligent migration management system, which consists of feature collection nodes, central databases, model training nodes, and policy configuration nodes. The system deploys an I/O log collector on each file storage server of EOS After filtering out irrelevant information, I/O logs are stored in the central key-value database in the format of <timestamp, parameter field, value>. Then the system integrates, normalizes, and batches the feature data according to the needs of different models, and writes to the online data queue for model training. The model training node is a multi-GPU computing cluster based on Tensorflow, caffe and other deep learning frameworks. The trained models and test results are stored in a distributed file system for persistent store and visible to all nodes. The policy configuration node scans the file list in the tape, mechanical hard disk, and solid state hard disk in the background, and performs the migration actions according to the model prediction results and the pre-set migration condition. Figure 2 System design Training a predictive model requires input of a large number of training samples. In general, in highenergy physics storage systems, file access is related to the computing mode of different users and different applications. In the Linux operating system, file operations are based on file descriptors and file pointers, including file creation, opening and closing, file read/write point movement, file reading and writing, etc. [13]. EOS's logging system provides historical access records in file name units: <timestamp, access category, file read and write bytes, file read/write point movements, file size>. These records can be organized into a multi-dimensional vector according to the type of operation, which is defined as the access feature vector of the file.  11. stdDevHourlySeeks: The mean square error seeks number in the read and write position of the file during a complete access during the hour.

Experiment results
In order to prevent the model from over-fitting, the training data is randomly divided into a training set (80%), a verification set (10%) and a test set (10%). A LSTM model is used to predict file access heat. Precision, recall and F-Score rate were used to evaluate the trained model. In large-scale data sets, precision and recall are generally mutually constrained. Precision is a description of random errors. It's the ratio of the correctly labeled files to the whole pool of files. Recall is the ratio of the correctly labeled hot files by our model to all who are hot in reality.
F-Score [14] comprehensively considers both precision and recall rate, and reconciles it at a certain ratio. F-Score is best if there is some sort of balance between precision and recall in the prediction system. The calculation formula is as follows: F − Score = (1 + ! ) * * ( ! * ) + Beta balances the importance of recall and precision. Beta was 1 in the calculation of F-Score. It means precision and recall were equally important to the test results. Since the ratio of the number of hot files and cold files in the training set is quite different, F-Score is used as the main indicator for model evaluation.

Conclusion and Outlook
In HEP storage, performance demands and data access imbalances in mass storage systems are increasing. As the amount of stored data grows, tiered storage requires data management software to migrate less active data to lower cost storage devices. Thus an automated data migration strategy is needed. This paper proposed a method of file access heat prediction. Data heat trend is used as the basis for migration to a relatively low-cost storage device. Due to the limitations of traditional models, it is difficult to achieve good results in predicting at such nonlinear scenes. This paper attempted to use the LSTM model to predict the evolution trend of data access heat. But until now it didn't consider impact brought by data migration to the storage performance. Next, the concept of migration cost will be introduced, which would consider impact on storage performance.