|
|
- 2017
海量GNSS小文件云存储优化方法研究
|
Abstract:
GNSS数据量呈指数级趋势增长，Hadoop分布式文件系统（HDFS）解决了海量GNSS数据存储瓶颈的难题，却面临内存占用多、文件相关性差和缺乏优化机制的问题。针对HDFS处理海量GNSS小文件效率不高的问题，结合GNSS数据类型、特点以及存储过程，提出了一种新的GNSS小文件云存储方法，优化了GNSS小文件的写入、读取、添加和删除策略。该方法分别按观测文件和解算成果的类型进行合并，对合并后的文件构建压缩Trie树索引，索引切分后，根据匹配算法分布式地存储索引块。实验采用国际GNSS服务（IGS）28 d的数据和产品进行云存储优化。结果表明，该方法降低了各节点内存消耗，提高了海量GNSS小文件写入、读取和删除的效率，实现了对海量GNSS小文件的高效云存储
| [1] | Li Linyang, Lv Zhiping, Chen Zhengsheng, et al. Cloud Storage Model of Massive Continuous Operating Reference Station System Data[J]. Journal of Navigation and Positioning, 2014, 2(3):64-70(李林阳, 吕志平, 陈正生, 等. 海量连续运行参考站网数据云存储模型[J]. 导航定位学报, 2014, 2(3):64-70) |
| [2] | Zhao Xiaoyong, Yang Yang, Sun Lili, et al. Metadata-Aware Small Files Storage Architecture on Hadoop[J]. Web Information Systems and Mining, 2012, 7529:136-143 |
| [3] | Dong Bo, Zheng Qinghua, Feng Tia, et al. An Optimized Approach for Storing and Accessing Small Files on Cloud Storage[J]. Journal of Network and Computer Applications, 2012, 35(6):1847-1862 |
| [4] | IGS Central Bureau. IGS Technical Report[OL]. http://kb.igs.org/hc/en-us/articles/115000200811-Technical-Report-2016, 2016 |
| [5] | Shi Chuang,Zhang Hongping,Gu Shengfeng,et al. Technology of Cloud Positioning and Its Platform for Positioning Service[J]. Geomatics and Information Science of Wuhan University, 2015, 40(8):995-999(施闯, 章红平, 辜声峰,等. 云定位技术及云定位服务平台[J]. 武汉大学学报·信息科学版, 2015, 40(8):995-999) |
| [6] | Wei Dai, Mostafa B. An Improved Task Assignment Scheme for Hadoop Running in the Clouds[J]. Journal of Cloud Computing, 2013, 2(1):1-16 |
| [7] | Cloudera. The Small Files Problem[EB/OL]. http://www.cloudera.com/blog/2009/02/the-small-files-problem.html, 2013 |
| [8] | Liu Jingnan, Liu Hui. Thought on Establishing a Continuous Operating Reference Station Network in China[J]. Geomatics and Information Science of Wuhan University, 2003, 28(5):27-31(刘经南, 刘晖. 建立我国卫星定位连续运行参考站网的若干思考[J]. 武汉大学学报·信息科学版, 2003, 28(5):27-31) |
| [9] | Liu Xiaojun, Xu Zhengquan, Pan Shaoming. A Massive Small File Storage Solution Combination of RDBMS and Hadoop[J]. Geomatics and Information Science of Wuhan University, 2013, 38(1):113-115(刘小俊, 徐正全, 潘少明.一种结合RDBMS和Hadoop的海量小文件存储方法[J].武汉大学学报·信息科学版, 2013, 38(1):113-115) |
| [10] | Xiong Lian, Xu Zhengquan, Wang Tao, et al. On the Store Strategy of Small Spatio-Temporal Data Files in Cloud Environment[J]. Geomatics and Information Science of Wuhan University, 2014, 39(10):1252-1256(熊炼, 徐正全, 王涛, 等. 云环境下的时空数据小文件存储策略[J].武汉大学学报·信息科学版, 2014, 39(10):1252-1256) |
| [11] | Liu Xuhui, Han Jizhong, Zhong Yunqin, et al. Implementing WebGIS on Hadoop:A Case Study of Improving Small File I/O Performance on HDFS[C]. IEEE International Conference on CLUSTER Computing and Workshops, New Orleans, USA,2009 |
| [12] | Maxime C, Thierry L. Encyclopedia of Database Systems[M].USA:Springer, 2009:3179-3182 |
| [13] | Feng Jianhua, Wang Jiannan, Li Guoliang. Trie-join:A Trie-based Method for Efficient String Similarity[J]. The VLDB Journal, 2012, 21(4):437-461 |
