分布式计算环境下的栅格数据存储策略

针对传统的栅格数据存储策略不能满足分布式计算环境下粗粒度数据访问需求，应对海量栅格数据计算时效率低下的问题，结合分布式文件系统的存储特点，同时考虑地图代数算子在Map/Reduce阶段以栅格瓦片为单位的计算特点，提出一种基于Hadoop分布式文件系统的栅格瓦片存储策略。围绕栅格数据瓦片分割、压缩瓦片数据组织与存储、分布式文件输入输出接口改进等方面对该存储策略加以实现，并使用基于该存储策略的地图代数局部算子的分布式计算流程加以验证。理论分析与实验结果表明，该策略能够显著提高分布式计算环境下空间分析算子的运算速度。
Traditional storage strategy of raster data cannot meet the demands of coarse-grained data processing under the distributed computing environment and has low efficiency when dealing with calculations for gigantic raster data. A storage strategy of raster tile data was presented on the basis of the storage characteristics of distributed file system. It also took the calculation characteristics of spatial analysis operators of map algebra into consideration, which uses raster tile as processing unit during map and reduce stage. The storage strategy was implemented by the following steps. Firstly raster data were divided into raster tiles. Then these tiles were compressed and organized by a special sequence in order to be transferred to Hadoop distributed file system. Finally input and output file interfaces were re-implemented to meet the data access requirements of map and reduce stage. The strategy was tested and verified by the distributed calculation process of local map algebra operators. Theoretical analysis and experimental results show that this strategy can significantly improve the processing speed of space analysis operators.