Globally distributed software defined storage (proposal)

The volume of the coming data in HEP is growing. The volume of the data to be held for a long time is growing as well. Large volume of data – big data – is distributed around the planet. The methods, approaches how to organize and manage the globally distributed data storage are required. The distributed storage has several examples for personal needs like own-cloud.org, pydio.com, seafile.com, sparkleshare.org. For enterprise-level there is a number of systems: SWIFT - distributed storage systems (part of Openstack), CEPH and the like which are mostly object storage. When several data center’s resources are integrated, the organization of data links becomes very important issue especially if several parallel data links between data centers are used. The situation in data centers and in data links may vary each hour. All that means each part of distributed data storage has to be able to rearrange usage of data links and storage servers in each data center. In addition, for each customer of distributed storage different requirements could appear. The above topics are planned to be discussed in data storage proposal.


Introduction
In tuning of real storage system there are a lot of aspects and questions even the storage servers are located in one room. The number of nuances is much more in geographically distributed storage systems. The part of them are obvious, for example, reliable and secure data transfer over Internet between data storage and client.
In addition there are data transfers to do data synchronization to guarantee the replicas of the data are completely equal each other in different sites. Storage of the data in remote site and data transfer over long distance do require additional precautions such as a data encryption. Data compression and data deduplication are also important to decrease the total data volume to transfer and store.
More and more often we are dealing with big data.

Current developments for distributed large volume data storage
A number of data storage studies were presented in [1][2][3]. Several developments are described in [4][5][6][7][8][9][10][11]. There are interesting benchmarks in [12,13]. Network architecture and Software Defined Network (SDN) approach, in particular, are discussed in [14]. Which are useful features to be included into the design of a scalable multi-tenant software defined storage (SDS) globally distributed system.

Main features of distributed storage
SDS is an evolving concept in which the management and provisioning of data storage is decoupled from the physical storage hardware. SDS should include: Automation -Simplified management that reduces the cost of maintaining the storage Infrastructure; Standard Interfaces -APIs for the management, provisioning and maintenance of storage devices and services; • Virtualized Data Path; • Block, File and Object interfaces that support applications written to these interfaces; • Scalability -Seamless ability to scale the storage infrastructure without disruption to availability or performance. Any multi-tenant applications running on the public cloud could benefit from the concepts introduced by SDS by managing the allocation of tenant data. Important features of the Globally Distributed Software Defined Storage (GDSDS): • Data store and Data transfer between client and data storage; also between components of the distributed storage; • Reliability: data replication, erasure coding; • Reduce the data volume to store and transfer: • Data compression, data deduplication (at number of replicas, at level of files, at block level) [15][16][17] • The preference of access to the geographically distributed data taking into account the context of CAP theorem [22], choose two of three options: consistency, availability, network partitioning.

General consideration
The general scheme of the GDSDS is shown in the figure 1. The clients can be also distributed around the World (client can be a person with desktop or smart phone, virtual machine in client data center, also client can be an organization with its own DC, etc). Several clients can use same VSI. In the figure 1 it is shown that all four clients have virtual storage which is located in two or more DCs. Obviously the connection between client and DCs with data located is very important for data transfer speed.
If VSI has one or more replicas it is good to allocate data replicas in different DCs. At this point we have to take into account several aspects. First of all we (and client) must plan the procedure of the replicas synchronization. The synchronization of the data between DCs takes time. Do we permit the clients to access the replica during synchronization time or we need to put client request in wait state until the synchronization is completed or clients have to read newest replica of data? The answer depends on client's business. In some cases clients are less sensitive to the consistency of the data, for example, in Google different clients in geographically different regions obtain different results on the same search request. In other situations clients could have another expectations, for example, they could expect exactly the same result (or very close in some context) on the request from any geographical region.  We had no problems with data loss in spite of many uncoordinated reboots, upgrades, reinstallations, etc. However clients can not perform some of operations to get data when CEPH cluster is in error state. Data transfer security with quantum key distribution model together with data encryption is in our nearest plans. Of course we are looking for support and volunteers.

Conclusion
We have analyzed a range of distributed storage systems, paid attention to most important features of these systems and suggested developing Geographically Distributed Software Defined Storage. It was found that it is better to use separate CEPH clusters for separate users especially with large storage requirements.