Dynamic Load Balancing Techniques for the Distributed Stream Processing Systems
分布式数据流处理系统的动态负载平衡技术

In the novel architecture for the large-scale distributed stream processing system, the whole system consists of a group of heterogeneous computer clusters. The whole system can achieve the global load balancing by balancing every cluster which consists of several homogeneous servers. The main goal of every cluster is exchanging the resources for the performance. In the cluster, enough servers are employed to get rid of the occurrence of overload phenomenon, so techniques for load shedding are not still necessary in the system. In the meanwhile, the number of active servers is decided by the practical load level and some servers can be put into the sleep mode for the sake of energy conservation when the load is rather low. The band-new initialization algorithm and dynamic load balancing algorithm are designed to accommodate the characteristic of increasing or decreasing the servers dynamically. Comparing to the traditional large-scale stream systems, these algorithms have better load balancing, lower complexity and faster response time because the number of servers in a single cluster is reduced sharply.