Optimization of YARN Hierarchical Resource Scheduling Algorithm
Abstract
We propose a hierarchical resource scheduling algorithm to solve the YARN resource problem, YARN is the resource scheduling management platform of Hadoop. Based on the resource scheduling DRF algorithm, the algorithm improves the fairness of the hierarchical resource scheduling by adjusting the calculation method of the usage rate of the parent node's primary resource. This paper establishes a hierarchical resource preemption model based on the characteristics of YARN task, and designs a framework to dynamically determine the preemptive task. The algorithm improves cluster resource’s utilization and rationality. The experimental results show our work can effectively shorten the overall run time and average YARN platform response time, improve the usage rate of the distributed cluster resources, and improve the Fair scheduling capability of YARN under the multiple queues and multiple resources.
DOI
10.12783/dtcse/csae2017/17493
10.12783/dtcse/csae2017/17493
Refbacks
- There are currently no refbacks.