XSKY SkyTeam storage solution builds an efficient data platform for autopilot.

With the continuous progress of autonomous driving technology, domestic pilot operation areas are blooming everywhere, but there are still many problems to be solved before autonomous driving is truly popularized. Among them, the maturity of autonomous driving decision-making system is one of the most critical issues.

The maturity of autonomous driving decision-making system needs to rely on the continuous collection of sufficient comprehensive data from road tests for model training, algorithm optimization and simulation in AI training system, so as to improve the coping ability in complex scenes and accelerate the realization of L4-level operational capability.

And build efficient autonomous driving? AI？ Training system, in addition to advanced algorithms and? GPU？ In addition to computing power, a data platform that carries massive data and interfaces with applications is equally important.

This paper describes how XSKY SkyTeam storage solution can help and serve the users of autonomous driving enterprises by introducing the workflow of autonomous driving AI training scene and its requirements for data platform.

1, autonomous driving? AI learning scene workflow

Autopilot? AI？ Training is responsible for data processing and conversion. The workflow includes data uploading, preprocessing, screening, labeling, cleaning and training. These steps will involve centralized storage, preprocessing (decryption, frame extraction, distortion elimination, etc.) of massive data. ), high-speed data flow between different storage systems, access control when docking with a third-party tag platform, and data transmission between different centers.

2. Challenges faced by data storage in autonomous driving AI learning system.

When the amount of data collected is increasing and the training efficiency needs to be improved, it will put forward higher requirements for the data platform of the infrastructure layer, which are mainly reflected in three aspects: first, the availability and cost optimization when mass storage is carried out; The second is the data interaction between the system and the platform; The third is to train the ultimate performance of linked storage.

Continuously optimize the availability and cost of the data platform when massive data expands.

Usually, users will have at least tens of petabytes of data and corresponding hundreds of billions of files. In this context, the intersection of flexible expansion of storage system, maximum support scale of cluster, high throughput of at least 10GB/s when uploading data, easy operation and maintenance, and optimization of storage cost are the challenges to storage.

Data cross-platform interaction requirements

Most users will adopt hybrid cloud? It? Architecture model, how to ensure the smooth flow of data between heterogeneous platforms, and how to achieve fine control of data permissions in data interaction with third-party annotation platforms will also become new challenges.

Requirements for storage efficiency in training sessions

Based on? K8S？ Distributed in? GPU？ The training mechanism can train millions of small files at a time, and the storage needs to provide high enough data throughput bandwidth and low latency to meet the efficiency requirements of upper computing power.

3. How does the 3.XSKY data storage scheme respond to the requirements of the scenario?

The core concept of XSKY SkyTeam for the autopilot AI scenario is: compatible with the mainstream business architecture of users, and the data platform is seamlessly connected with the upper application; In line with the future evolution direction of the business, the storage depth optimization meets the needs of the scene;

Smooth and compatible with mainstream business architecture

The infrastructure of many customers in the autonomous driving industry has changed from a public cloud model to a hybrid cloud model. Do you drive automatically when using the public cloud? AI？ Trained? Workflow? Most of them are built around the storage combination of "object storage+high performance file storage" to realize the automatic arrangement of business applications;

After the transition to the hybrid cloud model, what is the core content of the privatized data platform hosted by XSKY SkyTeam? Object storage? +? High-performance file storage, avoiding users? Workflow? Change, so as to reduce the repeated investment of the development side.

Storage availability to meet business scenarios

The availability of storage is reflected in flexible capacity expansion, unlimited data size, easy operation and maintenance, cross-platform capability and meeting the requirements of business applications for storage performance.

Flexible expansion, XSKY SkyTeam storage can support multi-mode expansion by node and cluster;

XSKY Trina background management system is simple to operate and maintain, providing visual interface, fine-grained alarm module and comprehensive monitoring ability of nodes and data;

Cross-platform capability, XSKY skyteam object management platform (XEOS)? Support docking with many mainstream public cloud storage at home and abroad to meet the requirements of data fluency. XSKY Trina Data Management System (X3DS) supports data replication and migration of heterogeneous platforms (such as reliable migration of user stock data);

In terms of performance, especially in the scene of "reading more and writing less" in the data training stage, the requirements for storage throughput and delay are very high. Can XSKY SkyTeam pass XGFS? Distributed file storage, or? Simpfennig? Xingfei all-flash storage all-in-one machine provides support, which not only meets the requirements? GPU？ Strict performance requirements for data extraction, and at the same time due to? XGFS？ And then what? Simpfennig? Is it the first one that can be supported in China? QLC？ Based on distributed storage, you can make full use of it? QLC？ Read and write characteristics and cost advantages, greatly reducing user deployment costs.

Multiple scene optimization to improve training efficiency.

Object storage? List? Performance optimization, through filtering and sorting action sinking, improve concurrency and other means. , reduce transmission and summary overhead, improve data extraction efficiency, and improve the stability of the cluster under high load;

XGFS？ Distributed file storage and integrity? NVMe？ what's up Simpfennig? All-in-one storage machine can be delivered by software or all-in-one storage machine, right? GPU？ Training courses provide high-performance file storage capacity;

In addition, there are many new functions such as independent metadata query service and open content processing framework, which can improve the business efficiency of data preprocessing and data screening.

Cost optimization of mass data storage

XSKY Trina Storage has the ability of data management in the whole life cycle of data, in which the storage classification and data compression functions can store data in multiple layers and freely flow in multiple pools according to the data's cold and hot. In addition, storage forms such as high-density nodes, Blu-ray magnetic storage integrated machine and tape archiving can greatly optimize the storage cost of users.

4. Scenario-oriented XSKY SkyTeam will continue to develop.

In the field of autonomous driving, the guarantee of training efficiency by storage platform and the cost optimization of mass storage will be a long-term theme. XSKY SkyTeam will continue to invest and introduce new features suitable for this scenario to help users of autonomous driving companies release data value more efficiently.