Research on Load Balancing in Task Allocation Process in Heterogeneous Hadoop Cluster

Abstract

The original task scheduling algorithm of Hadoop cannot meet the performance requirements of heterogeneous clusters. The existing Hadoop task scheduling algorithm assumes that the performance of each node is consistent. This algorithm can perform well in homogeneous clusters. However, in the heterogeneous Hadoop cluster, due to different performance between CPU, disks and memory, the load imbalance will occur in the cluster. In the view of the unbalanced load of Hadoop cluster in heterogeneous environment, this paper proposes a new algorithm named Load balancing algorithm based on heterogeneous environment (LBAHE), which takes into account the performance differences of each node. And when measuring the performance of nodes, the number of slots is no longer the only criterion. We also add the CPU, disk memory and other factors. Experiments show the efficiency of the new algorithm and it can perform tasks faster in heterogeneous clusters than original algorithm.