面向天河二号Lustre文件系统优化实践
Optimization Practice of Lustre File System for Tianhe No. 2

DOI: 10.12677/CSA.2021.1111264, PP. 2609-2618

Keywords: 文件系统，元数据，高可靠，优化
File System, Metadata, High Reliability, Optimization

Full-Text   Cite this paper   Add to My Lib

Abstract:

近年来，随着高性能计算技术的飞速发展，Lustre文件系统作为高性能计算重要组成部分越来越受到重视，对于Lustre文件系统元数据服务的高可靠性一直是研究的重点，针对Lustre文件系统元数据高可靠性的研究，本文设计了一种双MDT冗余的架构保证了元数据服务的高可靠性；同时随着应用复杂度的提高和计算规模的增加，对Lustre文件系统的性能要求也越来越高，特别是在处理海量小文件和I/O密集型应用时，对Lustre文件系统元数据的I/O性能提出了更高的要求，为了提高元数据的I/O性能，本文通过升级MDT元数据底层硬件设备来提升元数据的I/O性能和提升整体文件系统的I/O性能，满足新的应用对文件系统IO的要求。
In recent years, with the rapid development of high-performance computing technology, the Lustre file system has been paid more and more attention as an important component of high-performance computing. The high reliability of the metadata service of the Lustre file system has always been the focus of research. This paper has designed a dual MDT redundant architecture to ensure the high reliability of metadata services; at the same time, as the application complexity increases and the calculation scale increases, the performance requirements of the Lustre file system are also getting higher and higher, especially when dealing with massive small files and I/O intensive applications. I/O performance of the metadata of the Lustre file system has raised higher requirements. In order to improve the I/O performance of metadata, this article upgrades the underlying hardware equipment of MDT metadata to improve the metadata I/O performance and to improve the overall file system I/O performance to meet the new application requirements for file system IO.

References

[1]	Shu, Q., Qiao, F.L., Song, Z.Y. and Yin, X.Q. (2013) A Comparison of Two Global Ocean-Ice Coupled Models with Different Horizontal Resolutions. Acta Oceanologica Sinica, 32, 1-11. https://doi.org/10.1007/s13131-013-0335-z
[2]	宋振亚, 刘卫国, 刘鑫, 苏天赟, 刘海行, 尹训强. 海量数据驱动下的高分辨率海洋数值模式发展与展望[J]. 海洋科学进展, 2019, 37(2): 161-170.
[3]	Liang, J. and Nie, R.H. (2015) Lustre File System Based on Object Storage. Computer Engineering and Design, 36, 1666-1670.
[4]	Chen, Q., Chen, Z.N. and Jiang, J.H. (2014) MDDS: A Method to Improve the Metadata Performance of Parallel File System for HPC. Journal of Computer Research and Development, 51, 1663-1670.
[5]	Chen, J.-X. and Liu, X.-J. (2011) Analy-sis and Improvement of Distributed Replicated Block Device. Computer Engineering and Design, 32, 3599-3601, 3806.
[6]	Wang, H. and Sun, X.-Y. (2012) Application of Heartbeat and Drbd in Large-Capacity OLT. Modern Elec-tronics Technique, 35, 131-134, 137.
[7]	Li, Y. (2020) Research and Implementation of High Available Clusters Based on Linux. Computer Applications, 39, 35-38.
[8]	Gong, T.-N. and Zhou, S.-M. (2012) High Availability Cluster of Linux Based on DRBD. Computer and Information Technology, 20, 63-65.
[9]	Li, L.L., Wu, W.G. and Sun, L.X. (2012) Performance Optimization of Fine-Grained I/O in Parallel File System Lustre. Computer Engineering and Appli-cations, 48, 88-92.
[10]	Liu, G.M., Zou, D. and Zhang, C. (2009) Research on Lustre-Oriented Storage Acceleration with Solid State Disk. Journal of Computer Research and Development, 46, 371-375.
[11]	Mao, X.-F., Hou, X.-M. and Ma, H. (2016) VLBI Storage System of Computer Based on Disk Array. Computer Systems & Applications, 25, 107-111.
[12]	Li, Z., Zhou, E.Q. and Liao, X.K. (2009) Filter Cache: A Method for Improving I/O Performance of Lus-tre File System. Journal of Computer Research and Development, 46, 71-77.

Full-Text