Optimization for Data De-duplication Algorithm Based on Storage Environment Aware
基于存储环境感知的重复数据删除算法优化

Storage applications such as backup and archive are creating more and more duplication data, which has caused increasing waste in storage space and energy consumption, and how to delete duplication data has become a hot subject in research. CI}C(Content-Defined Chunking) is a prevail algorithm for data dcduplication and can be applicable in various environment,however it does not take into account some characteristics which are specific to individual environment and can influence its result. We studied the CDC' s application in storage system and put up with two constraints for determining parameters for CDC; (1) Determining parameters such as average block size based on the block organization of storage devices; (2)Determining block boundary based on candidate boundary distribution. The result indicates that, comparing with the separate CI}C without restraint conditions, these two constraints can achieve 16. 3% higher compression ratio with 4 data sets.