# Edge Intelligence Optimization for Large Language Model Inference with Batching and Quantization

arXiv: https://arxiv.org/abs/2405.07140 · HTML: https://mlsanity.com/paper/2405.07140/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 12 May 2024.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [Regulating ChatGPT and other Large Generative AI Models](https://mlsanity.com/paper/2302.02337/) (2023)
- [Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services](https://mlsanity.com/paper/2303.16129/) (2023)
- [Resource Allocation for Multiuser Edge Inference with Batching and Early Exiting (Extended Version)](https://mlsanity.com/paper/2204.05223/) (2022)

## Built on by (most cited first, 7 of 7)

- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directions](https://mlsanity.com/paper/2412.19823/) (2024)
- [PartialLoading: User Scheduling and Bandwidth Allocation for Parameter-sharing Edge Inference](https://mlsanity.com/paper/2503.22982/) (2025)
- [EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answering](https://mlsanity.com/paper/2511.19523/) (2025)
- [SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2602.01027/) (2026)
- [Multi-Agentic AI for Fairness-Aware and Accelerated Multi-modal Large Model Inference in Real-world Mobile Edge Networks](https://mlsanity.com/paper/2602.07215/) (2026)
- [Networking-Aware Energy Efficiency in Agentic AI Inference: A Survey](https://mlsanity.com/paper/2604.07857/) (2026)

## Abstract (arXiv)

Generative Artificial Intelligence (GAI) is taking the world by storm with its unparalleled content creation ability. Large Language Models (LLMs) are at the forefront of this movement. However, the significant resource demands of LLMs often require cloud hosting, which raises issues regarding privacy, latency, and usage limitations. Although edge intelligence has long been utilized to solve these challenges by enabling real-time AI computation on ubiquitous edge resources close to data sources, most research has focused on traditional AI models and has left a gap in addressing the unique characteristics of LLM inference, such as considerable model size, auto-regressive processes, and self-attention mechanisms. In this paper, we present an edge intelligence optimization problem tailored for LLM inference. Specifically, with the deployment of the batching technique and model quantization on resource-limited edge devices, we formulate an inference model for transformer decoder-based LLMs. Furthermore, our approach aims to maximize the inference throughput via batch scheduling and joint allocation of communication and computation resources, while also considering edge resource constraints and varying user requirements of latency and accuracy. To address this NP-hard problem, we develop an optimal Depth-First Tree-Searching algorithm with online tree-Pruning (DFTSP) that operates within a feasible time complexity. Simulation results indicate that DFTSP surpasses other batching benchmarks in throughput across diverse user settings and quantization techniques, and it reduces time complexity by over 45% compared to the brute-force searching method.