|
|
- 2017
通信密集环境下基于内存利用率的预计算方法
|
Abstract:
针对通信密集型图计算环境下原静态最大消息数阈值方法由于内存不足导致的频繁低效I/O问题,提出了一种基于内存利用率的预计算方法。该方法利用了图应用的计算满足交换律和结合律的特点,根据当前进程的内存利用率判断是否将本轮超步通信过程中的部分消息进行预计算,同时在预计算过程中使用细粒度锁以增大预计算线程的并发度;在下轮超步的正常计算时合并上轮的预计算结果,实现了通信和计算的重叠,达到减少作业响应时间和磁盘I/O开销的目的。实验结果表明,在通信密集场景下,该方法在性能和I/O开销上均优于已有的MMT方法,作业响应时间减少了5.9%~79.0%,同时计算过程中的磁盘开销减少了9.99％～79.87％。
A precomputation method based on memory utilization (MUP) is proposed to improve the I/O？？inefficient problem caused by limited memory of the static maximum message threshold method (MMT) in intensive communication environments. The method leverages both the associative law and commutative law of graph computations, and precomputes some partial incoming messages of current super？？step based on the process memory utilization. The precomputed results are then combined in the next super？？step’s normal computations. Moreover, a find？？grained lock mechanism is used to increase the parallel granularity of precomputation threads. The method reduces the job response time and the expensive disk I/O costs incurred by messages through overlapping computation and communication. Experimental results show that MUP is better than the original MMT method in both the performance and I/O costs. The job response time is improved by 5.9%~79.0% and high redundant disk I/O costs are reduced by 9.99％~79.87％ in intensive communication environments
| [1] | ［3］ROY A, MIHAILOVIC I, ZWAENEPOEL W. X？？stream: edge？？centric graph processing using streaming partitions ［C］∥Proceedings of the 24th ACM Symposium on Operating Systems Principles. New York, USA: ACM, 2013: 472？？488. |
| [2] | ［7］罗乐, 刘轶, 钱德沛. 内存计算技术研究综述 ［J］. 软件学报, 2016, 27(8): 2147？？2167. |
| [3] | ［13］VALIANT L G. A bridging model for parallel computation ［J］. Communications of the ACM, 1990, 33(8): 103？？111. |
| [4] | ［11］MCCUNE R R, WENINGER T, MADEY G. Thinking like a vertex: a survey of vertex？？centric frameworks for large？？scale distributed graph processing ［J］. ACM Computing Surveys, 2015, 48(2): 1？？39. |
| [5] | ［14］ZHANG Yanfeng, GAO Qixin, GAO Lixin, et al. Maiter: an asynchronous graph processing framework for delta？？based accumulative iterative computation ［J］. IEEE Transactions on Parallel and Distributed Systems, 2014, 25(8): 2091？？2100. |
| [6] | ［16］GONZALEZ J E, XIN R S, DAVE A, et al. GraphX: graph processing in a distributed dataflow framework ［C］∥Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation. Berkeley, CA, USA: USENIX Association, 2014: 599？？613. |
| [7] | LUO Le, LIU Yi, QIAN Depei. Survey on in？？memory computing technology ［J］. Journal of Software, 2016, 27(8): 2147？？2167. |
| [8] | ［8］WANG Zhigang, GU Yu, BAO Yubin, et al. Hybrid pulling/pushing for I/O？？efficient distributed and iterative graph computing ［C］∥Proceedings of the 47th International Conference on Management of Data. New York, USA: ACM, 2016: 479？？494. |
| [9] | ［9］KYROLA A, BLELLOCH G, GUESTRIN C. GraphChi: large？？scale graph computation on just a PC ［C］∥Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation. Berkeley, CA, USA: USENIX Association, 2012: 31？？46. |
| [10] | ［10］BU Yingyi, BORKAR V, JIA Jianfeng, et al. Pregelix: big(ger) graph analytics on a dataflow engine ［J］. Proceedings of the VLDB Endowment, 2014, 8(2): 161？？172. |
| [11] | ［12］SHUN Julian, DHULIPALA L, BLELLOCH G E. Smaller and faster: parallel processing of compressed graphs with Ligra+ ［C］∥Proceedings of the 25th Data Compression Conference. Piscataway, NJ, USA: IEEE Computer Society, 2015: 403？？412. |
| [12] | ［15］SHIRAZI J. Java performance tuning ［M］. Sebastopol, CA, USA: O’Reilly ＆ Associates Inc., 2000: 45？？49. |
| [13] | ［1］CHING A, EDUNOV S, KABILJO M, et al. One trillion edges: graph processing at facebook？？scale ［J］. Proceedings of the VLDB Endowment, 2015, 8(12): 1804？？1815. |
| [14] | ［2］MALEWICZ G, AUSTERN M H, BIK A J C, et al. Pregel: a system for large？？scale graph processing ［C］∥Proceedings of the 41st ACM Sigmod International Conference on Management of Data. New York, USA: ACM, 2010: 135？？146. |
| [15] | ［4］GONZALEZ J E, LOW Yucheng, GU Haijie, et al. PowerGraph: distributed graph？？parallel computation on natural graphs ［C］∥Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation. Berkeley, CA, USA: USENIX Association, 2012: 17？？30. |
| [16] | ［5］SHI Xuanhua, CHEN Ming, HE Ligang, et al. Mammoth: gearing hadoop towards memory？？intensive mapreduce applications ［J］. IEEE Transactions on Parallel and Distributed Systems, 2015, 26(8): 2300？？2315. |
| [17] | ［6］DHIMAN G, AYOUB R, ROSING T. PDRAM: a hybrid PRAM and DRAM main memory system ［C］∥Proceedings of the 46th Annual Design Automation Conference. New York, USA: ACM, 2009: 664？？669. |
