# Scalability Optimization in Cloud-Based AI Inference Services: Strategies for Real-Time Load Balancing and Automated Scaling

arXiv: https://arxiv.org/abs/2504.15296 · HTML: https://mlsanity.com/paper/2504.15296/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Builds on (most cited first, 12 of 12)

- [AD-LLM: Benchmarking Large Language Models for Anomaly Detection](https://mlsanity.com/paper/2412.11142/) (2024)
- [Confidence Trigger Detection: Accelerating Real-time Tracking-by-detection Systems](https://mlsanity.com/paper/1902.00615/) (2019)
- [CarbonScaler: Leveraging Cloud Workload Elasticity for Optimizing Carbon-Efficiency](https://mlsanity.com/paper/2302.08681/) (2023)
- [NLP-ADBench: NLP Anomaly Detection Benchmark](https://mlsanity.com/paper/2412.04784/) (2024)
- [Deep Learning based Multi-Label Image Classification of Protest Activities](https://mlsanity.com/paper/2301.04212/) (2023)
- [T-GAE: Transferable Graph Autoencoder for Network Alignment](https://mlsanity.com/paper/2310.03272/) (2023)
- [Few-shot Name Entity Recognition on StackOverflow](https://mlsanity.com/paper/2404.09405/) (2024)
- [Prediction of COVID-19 Patients' Emergency Room Revisit using Multi-Source Transfer Learning](https://mlsanity.com/paper/2306.17257/) (2023)
- [Influence Pathway Discovery on Social Media](https://mlsanity.com/paper/2309.16071/) (2023)
- [Deep Learning for Content-based Personalized Viewport Prediction of 360-Degree VR Videos](https://mlsanity.com/paper/2003.00429/) (2020)
- [Decentralized Edge-to-Cloud Load-balancing: Service Placement for the Internet of Things](https://mlsanity.com/paper/2005.00270/) (2020)
- [Fed-LDR: Federated Local Data-infused Graph Creation with Node-centric Model Refinement](https://mlsanity.com/paper/2411.04936/) (2024)

## Built on by (most cited first, 10 of 10)

- [Cloud-Based AI Systems: Leveraging Large Language Models for Intelligent Fault Detection and Autonomous Self-Healing](https://mlsanity.com/paper/2505.11743/) (2025)
- [Cross-Cloud Data Privacy Protection: Optimizing Collaborative Mechanisms of AI Systems by Integrating Federated Learning and LLMs](https://mlsanity.com/paper/2505.13292/) (2025)
- [Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation](https://mlsanity.com/paper/2505.17391/) (2025)
- [DeepRAG: Integrating Hierarchical Reasoning and Process Supervision for Biomedical Multi-Hop QA](https://mlsanity.com/paper/2506.00671/) (2025)
- [Credit Risk Analysis for SMEs Using Graph Neural Networks in Supply Chain](https://mlsanity.com/paper/2507.07854/) (2025)
- [Federated Learning-Based Data Collaboration Method for Enhancing Edge Cloud AI System Security Using Large Language Models](https://mlsanity.com/paper/2506.18087/) (2025)
- [QuickMerge++: Fast Token Merging with Autoregressive Prior](https://mlsanity.com/paper/2508.13204/) (2025)
- [Leveraging Large Language Model for Intelligent Log Processing and Autonomous Debugging in Cloud AI Platforms](https://mlsanity.com/paper/2506.17900/) (2025)
- [Cognitive Load Traces as Symbolic and Visual Accounts of Deep Model Cognition](https://mlsanity.com/paper/2510.15980/) (2025)
- [User-Assisted Collaborative Distributed Inference for Efficient QoS-Aware Autoscaling](https://mlsanity.com/paper/2608.11840/) (2026)

## Abstract (arXiv)

The rapid expansion of AI inference services in the cloud necessitates a robust scalability solution to manage dynamic workloads and maintain high performance. This study proposes a comprehensive scalability optimization framework for cloud AI inference services, focusing on real-time load balancing and autoscaling strategies. The proposed model is a hybrid approach that combines reinforcement learning for adaptive load distribution and deep neural networks for accurate demand forecasting. This multi-layered approach enables the system to anticipate workload fluctuations and proactively adjust resources, ensuring maximum resource utilisation and minimising latency. Furthermore, the incorporation of a decentralised decision-making process within the model serves to enhance fault tolerance and reduce response time in scaling operations. Experimental results demonstrate that the proposed model enhances load balancing efficiency by 35 and reduces response delay by 28, thereby exhibiting a substantial optimization effect in comparison with conventional scalability solutions.