Review of big data stream processing: latency optimization and dynamic resource management algorithms
Article Sidebar
Main Article Content
Abstract
This review paper systematically examines modern architectures, algorithms, and approaches aimed at reducing latency and dynamically managing resources in real-time big data stream processing systems. The characteristics of Lambda, Kappa, and hybrid architectures, cloud-native platforms, as well as Edge–Fog–Cloud hybrid environments are analyzed. Network-level and computation-level latency factors, along with mitigation techniques – including operator placement, task offloading, RDMA technology, and machine learning-based prediction models – are investigated. Dynamic resource management issues in cloud environments are addressed, including reactive, proactive, and hybrid auto-scaling algorithms, time series-based approaches, deep learning and reinforcement learning methods, and multi-objective task scheduling. The analysis demonstrates that the highest efficiency is achievable through the integration of complementary approaches – proactive forecasting, hybrid auto-scaling, and ML-based decision-making. Open research challenges and promising future directions are identified.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
J. Dean, S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Proc. 6th Symp. on Operating System Design and Implementation (OSDI), San Francisco, CA, 2004.
M. Armbrust et al., "A View of Cloud Computing," Communications of the ACM, vol. 53, no. 4, pp. 50-58, 2010.
T. Lorido-Botran, J. Miguel-Alonso, J.A. Lozano, "A Review of Auto-scaling Techniques for Elastic Applications in Cloud Environments," Journal of Grid Computing, vol. 12, pp. 559-592, 2014.
G. Ananthanarayanan et al., "Scarlett: Coping with Skewed Content Popularity in MapReduce Clusters," Proc. EuroSys 2011, Salzburg, Austria, 2011.
L. Kleinrock, Queueing Systems, Volume 1: Theory. New York: John Wiley & Sons, 1975.
C. Colarusso et al., "PROMENADE: A big data platform for handling city complex networks with dynamic graphs," Future Generation Computer Systems, vol. 137, pp. 129-145, 2022.
M. Francia, M. Golfarelli, M. Pasini, "Process-driven design of cloud data platforms," Information Systems, vol. 131, p. 102527, 2025.
A. Roukh et al., "Big Data Processing Architecture for Smart Farming," Procedia Computer Science, vol. 177, pp. 78-85, 2020.
S. Nastic, H. Truong, S. Dustdar, "Latency-aware placement of stream processing operators in modern-day stream processing frameworks," Future Generation Computer Systems, vol. 95, pp. 208-221, 2019.
M.M. Rathore, A. Ahmad, A. Paul, G. Jeon, "A big data platform for smart cities," Future Generation Computer Systems, vol. 101, pp. 101-112, 2019.
A.I. Maarouf, S. Tata, L. Bellatreche, "Process-driven design of cloud data platforms," Future Generation Computer Systems, vol. 109, pp. 571-586, 2020.
A. Zanella et al., "A new Kappa architecture for IoT data management in smart farming," Computer Networks, vol. 168, 2020.
I. Afolayan, A.F.S. Dehghantanha, K.-K.R. Choo, "Big data analytics in telecommunications: Governance, architecture and use cases," J. Network and Computer Applications, vol. 154, 2020.
Y. Liu et al., "Beyond 5G: PHWAN - A secure, low-latency, and cost-effective framework for Industry 4.0 smart manufacturing," Computer Networks, vol. 234, 2023.
Y. Wang, Z. Liu, H. Zhang, "Network performance evaluation criterion model for low latency in industrial 50G-PON network," Optical Fiber Technology, vol. 82, 2024.
S. Kar, P. Mishra, K.-C. Wang, "Dynamic packet duplication for reliable low latency communication in 5G NR-DC networks," Computer Networks, vol. 234, 2023.
Y. Zhang, M. Li, K. Zhou, K. You, Y. Yu, "Design and implementation of an RDMA-based data transmission system prototype," Nuclear Engineering and Technology, vol. 58, 2026.
M.M. Hassan et al., "ML WPStreamCloud: ML-based workload prediction and task clustering for efficient stream application offloading," J. Network and Computer Applications, vol. 201, 2022.
H. Zhang, Y. Li, X. Chen, "An adaptive hybrid edge-cloud collaborative offloading method for large-scale computational tasks," Future Generation Computer Systems, vol. 148, pp. 48-61, 2023.
A. Alqahtani, A. Alabdulatif, A. Alenezi, "Metaheuristic-optimized forecasting in a smart edge-fog-cloud energy management framework," Energy Reports, vol. 10, pp. 1120-1133, 2024.
M. Abdelbaky, J. Diaz-Montes, M. Parashar, M. Sadjadi, "Towards resource-efficient reactive and proactive auto-scaling for microservice architectures," Journal of Systems and Software, vol. 188, 2022.
J.V.V. Sobral, P.A.D. Domingos, L. Veiga, "Enhancing the output of time series forecasting algorithms for cloud resource provisioning," Future Generation Computer Systems, vol. 152, pp. 351-365, 2025.
M.A. Tawfeek et al., "An efficient and autonomous dynamic resource allocation in cloud computing with optimized task scheduling," Future Generation Computer Systems, vol. 32, pp. 232-246, 2014.
R. Singh, P.K. Gupta, "VBDPA: A multi-criteria task scheduling algorithm in container based cloud computing environment," Journal of Cloud Computing, vol. 11, no. 54, 2022.
X. Li, Z. Zhao, F. Zhao, H. Zhang, "Multi-objective task offloading optimization using deep reinforcement learning with resource distribution clustering," Applied Soft Computing, vol. 143, 2023.
S. Sotiriadis, N. Bessis, E. Asimakopoulou, "Dataset on resource allocation and usage for a private cloud," Data in Brief, vol. 40, 2022.
A.A. Alqahtani, A. Alabdulatif, A. Alenezi, "Optimizing multi-time series forecasting for enhanced cloud resource utilization," Future Generation Computer Systems, vol. 154, pp. 380-394, 2024.
R. Anwar, M. Rehman, S.K. Das, "A classification framework for straggler mitigation and management in a heterogeneous Hadoop cluster," J. Parallel and Distributed Computing, vol. 170, pp. 41-55, 2022.
S. Guo, B. Xiao, Y. Yang, "DRL-based routing algorithm with guaranteed loss, latency and bandwidth in SDN networks," Computer Networks, vol. 234, 2023.
J. Li, J. Wu, Z. Zhang, "A Petri Net-based framework for modeling and simulation of resource scheduling policies in Edge Cloud Continuum," Simulation Modelling Practice and Theory, vol. 132, 2025.
D. Monaco et al., "Real-time latency prediction for cloud gaming applications," Computer Networks, vol. 264, 2025.
G. Zhou, W. Tian, R. Buyya et al., "Deep reinforcement learning-based methods for resource scheduling in cloud computing: A review and future directions," Artificial Intelligence Review, vol. 57, p. 124, 2024.
H. Hou, S.N.A. Jawaddi, A. Ismail, "Energy efficient task scheduling based on deep reinforcement learning in cloud environment," Future Generation Computer Systems, vol. 151, pp. 214-231, 2024.
M.U. Demirezen, T.S. Navruz, "Performance Analysis of Lambda Architecture-Based Big-Data Systems on Air/Ground Surveillance Application with ADS-B Data," Sensors, vol. 23, no. 17, p. 7580, 2023.
A. da Silva Veith et al., "Latency-Aware Strategies for Deploying Data Stream Processing Applications on Large Cloud-Edge Infrastructure," IEEE Transactions on Parallel and Distributed Systems, 2021.