|
|
- 2015
一种短作业环境下的延迟调度算法
|
Abstract:
针对短作业场景下YARN平台中延迟调度算法基于静态时间等待阈值,不能进行合理等待的问题,提出了一种云计算环境中基于本地性资源预测的延迟调度算法(locality resource forecast delay scheduling, LRFD)。该算法综合考虑短作业和资源可用性动态变化的特点进行任务调度,根据节点上任务的完成进度和作业未处理数据在集群中的分布状况预估作业的本地性资源信息,从而判断是否需要进行等待以提高系统性能,实现了对本地性资源的合理等待。实验结果表明:在短作业场景下,LRFD算法的性能和稳定性均优于已有的延迟算法,作业性能平均提升约10％,最大加速比可达3倍以上。
A delay scheduling algorithm based on locality resource forecasting(LRFD) is proposed to address the unreasonable waiting problem generalized by the static time？？wait threshold in delay scheduling algorithm of YARN platform for short jobs. The algorithm takes both the characteristics of short jobs and dynamic resource availability into consideration to assign tasks. It estimates local resources to make reasonable waiting according to both the task progress on the nodes and the unhandled splits distribution in the cluster of the job. Experimental results and comparison with the traditional delay scheduling algorithm show that LRFD gets a better stability and improves the performance about 10% for short jobs on average and achieves a maximum speedup up to three times
| [1] | ［6］HENDERSON R L. Job scheduling under the portable batch system ［C］∥Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing. Berlin, Germany, : Springer？？Verlag, 1995: 279？？294. |
| [2] | ［7］FREY J, TANNENBAUM T, LIVNY M, et al. Condor？？G: a computation management agent for multi？？institutional grids ［J］. Cluster Computing, 2002, 5(3): 237？？246. |
| [3] | ［8］HINDMAN B, KONWINSKI A, ZAHARIA M, et al. Mesos: a platform for fine？？grained resource sharing in the data center ［C］∥Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA, USA: USENIX Association, 2011: 295？？308. |
| [4] | ［9］VAVILAPALLI V K, MURTHY A C, DOUGLAS C, et al. Apache hadoop yarn: yet another resource negotiator ［C］∥Proceedings of the 4th Annual Symposium on Cloud Computing. New York, NY, USA: ACM, 2013: 1？？16. |
| [5] | ［10］ZAHARIA M, BORTHAKUR D, SARMA J S, et al. Job scheduling for multi？？user Mapreduce clusters［R］. Berkeley, California, USA: EECS Department, University of California, Berkeley, 2009: 1？？16. |
| [6] | ［11］ZAHARIA M, BORTHAKUR D, SEN SARMA J, et al. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling ［C］∥Proceedings of the 5th European Conference on Computer Systems, New York, NY, USA: ACM, 2010: 265？？278. |
| [7] | ［12］ELMELEEGY K. Piranha: optimizing short jobs in hadoop ［J］. Proceedings of the Very Large Data Base Endowment, 2013, 6(11): 985？？996. |
| [8] | ［2］CHEN Y, ALSPAUGH S, KATZ R. Interactive analytical processing in big data systems: a cross？？industry study of MapReduce workloads ［J］. Proceedings of the Very Large Data Base Endowmat, 2012, 5(12): 1802？？1813. |
| [9] | ［3］ISARD M, PRABHAKARAN V, CURREY J, et al. Quincy: fair scheduling for distributed computing clusters ［C］ ∥Proceedings of the ACM SIGOPS 22nd |
| [10] | Symposium on Operating Systems Principles. New York, NY, USA: ACM, 2009: 261？？276. |
| [11] | ［4］REN Z, XU X, WAN J, et al. Workload analysis, implications, and optimization on a production hadoop cluster: a case study on Taobao ［J］. IEEE Transactions on Services Computing, 2014, 7(2): 307？？321. |
| [12] | ［13］ZAHARIA M, KONWINSKI A, JOSEPH A D, et al. Improving MapReduce performance in heterogeneous environments ［C］∥Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation. Berkeley, CA, USA: USENIX Association, 2008: 29？？42. |
| [13] | ［1］DEAN J, GHEMAWAT S. MapReduce: simplified data processing on large clusters ［J］. Communications of the ACM, 2008, 51(1): 107？？113. |
| [14] | ［5］ZAHARIA M, CHOWDHURY M, FRANKLIN M J, et al. Spark: cluster computing with working sets ［C］∥Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing. Berkeley, CA, USA: USENIX Association, 2010: 10. |
