Modeling and Solution for Load Balancing Optimization in Distributed Stream Processing System Management

doi:10.12005/orms.2021.0124

Operations Research and Management Science ›› 2021, Vol. 30 ›› Issue (4): 155-162.DOI: 10.12005/orms.2021.0124

• Application Research • Previous Articles Next Articles

Modeling and Solution for Load Balancing Optimization in Distributed Stream Processing System Management

TANG Ying-feng¹, CHEN Shi-ping²

1. Academic Affairs Section, Shanghai University of International Business and Economics, Shanghai 201620, China;
2. Management School, University of Shanghai for Science and Technology, Shanghai 200093, China

Received:2019-01-02 Online:2021-04-25

分布式数据流处理系统管理中负载均衡问题建模与求解

唐颖峰¹, 陈世平²

1.上海对外贸经贸大学教务处,上海 201620;
2.上海理工大学管理学院,上海 200093

作者简介:唐颖峰(1983-),男,甘肃嘉峪关人,助理研究员,博士,主要研究方向为云计算、数据挖掘;陈世平(1964-),男,教授,博士生导师,博士,主要研究方向为云计算、计算机网络通信、信息检索。
基金资助:
国家自然科学基金资助项目(61170277,61472256);上海市教委科研创新重点项目(12zz137);上海市一流学科建设项目(S1201YLXK)

Abstract

Abstract: The paper concentrates on the load balancing problem in the management of distributed stream processing system. The operation mechanism of distributed stream processing system and the causes of unbalanced load of nodes is expounded, and an optimization scheme for load balancing adjustment is proposed in the paper. The model of the proposed optimization scheme is then established, and the applicable conditions of the model are theoretically analyzed. Then the model is solved with ant colony optimization, and the algorithm is then improved to meet the real-time requirement of distributed stream processing system.Finally, the validity of the model and its solution in solving the node load balancing problem in the management of distributed stream processing system is proved by experiments.

Key words: system management, distributed stream processing system, load balancing;combinatorial optimization problem, ant colony optimization

摘要： 对分布式数据流处理系统管理中,处理节点负载均衡问题进行了研究。阐述了分布式数据流处理系统的运行机理以及节点负载不均衡的成因,并提出了对系统负载均衡调整的优化方案;对提出的优化方案建立模型,并对模型的适用条件进行理论分析;然后采用蚁群算法对模型进行求解,并针对分布式数据流处理系统实时性的需求对算法进行改进;最后用实验证明本文所建立的模型及其求解方法对于解决分布式数据流处理系统管理中节点负载均衡问题的有效性。

关键词: 系统管理, 分布式数据流处理系统, 负载均衡, 组合优化问题, 蚁群算法

CLC Number:

C931

TANG Ying-feng, CHEN Shi-ping. Modeling and Solution for Load Balancing Optimization in Distributed Stream Processing System Management[J]. Operations Research and Management Science, 2021, 30(4): 155-162.

唐颖峰, 陈世平. 分布式数据流处理系统管理中负载均衡问题建模与求解[J]. 运筹与管理, 2021, 30(4): 155-162.

References

[1] Zhang J,Wang F Y,Wang K,et al.Data-driven intelligent transportation systems: a survey[J].IEEE Transactions on Intelligent Transportation Systems,2011,12(4): 1624-1639.
[2] 余辰,张丽娟,金海.大数据驱动的智能交通系统研究进展与趋势[J].物联网学报,2018,2(1):56-63.
[3] Kwon Y C,Balazinska M,Greenberg A G.Fault-tolerant stream processing using a distributed,replicated file system[J].Proceedings of the Vldb Endowment,2008,1(1): 574-585.
[4] Drougas Y,Repantis T,Kalogeraki V.Load balancing techniques for distributed stream processing applications in overlay environments[C]//IEEE International Symposium on Object &Component-oriented Real-time Distributed Computing.2006.
[5] Zhou Y,Ooi B C,Tan K L.Dynamic load management for distributed continuous query systems[C]//International Conference on Data Engineering.IEEE,2005.
[6] Xing Y,Zdonik S,Hwang J H.Dynamic load distribution in the borealis stream processor[C]//International Conference on Data Engineering.IEEE Computer Society,2005.
[7] Tatbul N,Zdonik S.Dealing with overload in distributed stream processing systems[C]//International Conference on Data Engineering Workshops.IEEE,2006.
[8] 邓华锋,刘云生,肖迎元.分布式数据流处理系统的动态负载平衡技术[J].计算机科学,2007,34(7):120-123.
[9] Shah M A,Chandrasekaran S,et al.Flux: an adaptive partitioning operator for continuous query systems[C]//International Conference on Data Engineering.IEEE,2003.
[10] Gedik B.Partitioning functions for stateful data parallelism in stream processing[J].The VLDB Journal,2014,23(4): 517-539.
[11] Dorigo M,Stützle T.Ant colony optimization: overview and recent advances[J].Handbook of Metaheuristics,2010,146(5): 227-263.
[12] 吕国亮.集合的划分与第二类Stirling数[J].渭南师范学院学报,2005,20(2):22-24.
[13] 罗飞,任强,丁炜超,卢海峰.基于最小松弛量的启发式一维装箱算法[J].计算机科学,2019,46(9):315-320.
[14] 马良,项培军.蚂蚁算法在组合优化中的应用[J].管理科学学报,2001,4(2):32-37.
[15] 唐颖峰,陈世平.一种基于网格块的分布式数据流聚类算法[J].小型微型计算机系统,2016,37(3):488-493.

Modeling and Solution for Load Balancing Optimization in Distributed Stream Processing System Management

分布式数据流处理系统管理中负载均衡问题建模与求解

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 5

Recommended Articles

Metrics

[1]	MENG Qiu-nan, BAI Xue, ZHAO Cong. Batch Production Planning for Mixed Flow of No-wait Assembly Lines [J]. Operations Research and Management Science, 2017, 26(3): 178-186.
[2]	DING Qiu-lei, HU Xiang-pei, JIANG Yang, RUAN Junhu. Disruption Management Model for Location Change in Express Delivery [J]. Operations Research and Management Science, 2017, 26(12): 46-52.
[3]	CAI Wan-jun, WANG Chen-yu, YU Bin, YANG Zhong-zhen, YAO Bao-zhen. Improved Ant Colony Algorithm for Period Vehicle Routing Problem [J]. Operations Research and Management Science, 2014, 23(5): 70-77.
[4]	HU Chong-hai, LI Yi-jun, JIANG Wei, WANG Tie-jun. Dynamic Continuous Ant Colony Optimization and Its Application to Space-based Warning System [J]. Operations Research and Management Science, 2011, 20(2): 89-96.
[5]	GUO Hai-xiang, YANG Juan, MA Zheng-yan, LI Lan-lan. Optimizing Mine Materials Heterogeneous-vehicle Distribution by Improved Genetic Algorithm [J]. Operations Research and Management Science, 2011, 20(2): 193-199.