# Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models

arXiv: https://arxiv.org/abs/2401.00625 · HTML: https://mlsanity.com/paper/2401.00625/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jan 2024.

## Built on by (most cited first, 30 of 67)

- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [Uncertainty Quantification for In-Context Learning of Large Language Models](https://mlsanity.com/paper/2402.10189/) (2024)
- [Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency](https://mlsanity.com/paper/2504.03360/) (2025)
- [Fast yet Safe: Early-Exiting with Risk Control](https://mlsanity.com/paper/2405.20915/) (2024)
- [R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?](https://mlsanity.com/paper/2410.05474/) (2024)
- [SparseLLM: Towards Global Pruning for Pre-trained Language Models](https://mlsanity.com/paper/2402.17946/) (2024)
- [MVGamba: Unify 3D Content Generation as State Space Sequence Modeling](https://mlsanity.com/paper/2406.06367/) (2024)
- [Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey](https://mlsanity.com/paper/2411.10478/) (2024)
- [The Rise of Small Language Models in Healthcare: A Comprehensive Survey](https://mlsanity.com/paper/2504.17119/) (2025)
- [Abstractive Text Summarization: State of the Art, Challenges, and Improvements](https://mlsanity.com/paper/2409.02413/) (2024)
- [Energy-Latency Manipulation of Multi-modal Large Language Models via Verbose Samples](https://mlsanity.com/paper/2404.16557/) (2024)
- [Pruning as a Domain-specific LLM Extractor](https://mlsanity.com/paper/2405.06275/) (2024)
- [LLM-Oriented Token-Adaptive Knowledge Distillation](https://mlsanity.com/paper/2510.11615/) (2025)
- [ELAD: Explanation-Guided Large Language Models Active Distillation](https://mlsanity.com/paper/2402.13098/) (2024)
- [Persistent Topological Features in Large Language Models](https://mlsanity.com/paper/2410.11042/) (2024)
- [AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation](https://mlsanity.com/paper/2508.00733/) (2025)
- [DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving](https://mlsanity.com/paper/2408.16647/) (2024)
- [RARE: Retrieval-Augmented Reasoning Modeling](https://mlsanity.com/paper/2503.23513/) (2025)
- [Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing](https://mlsanity.com/paper/2409.18286/) (2024)
- [Neural Brain: A Neuroscience-inspired Framework for Embodied Agents](https://mlsanity.com/paper/2505.07634/) (2025)
- [LatentLLM: Attention-Aware Joint Tensor Compression](https://mlsanity.com/paper/2505.18413/) (2025)
- [Can LLMs Understand Social Norms in Autonomous Driving Games?](https://mlsanity.com/paper/2408.12680/) (2024)
- [Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques](https://mlsanity.com/paper/2506.06579/) (2025)
- [Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures](https://mlsanity.com/paper/2604.09048/) (2026)
- [A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design](https://mlsanity.com/paper/2607.09084/) (2026)
- [Achieving Peak Performance for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2409.04833/) (2024)
- [From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models](https://mlsanity.com/paper/2409.18996/) (2024)

## Abstract (arXiv)

The burgeoning field of Large Language Models (LLMs), exemplified by sophisticated models like OpenAI's ChatGPT, represents a significant advancement in artificial intelligence. These models, however, bring forth substantial challenges in the high consumption of computational, memory, energy, and financial resources, especially in environments with limited resource capabilities. This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs. We categorize methods based on their optimization focus: computational, memory, energy, financial, and network resources and their applicability across various stages of an LLM's lifecycle, including architecture design, pretraining, finetuning, and system design. Additionally, the survey introduces a nuanced categorization of resource efficiency techniques by their specific resource types, which uncovers the intricate relationships and mappings between various resources and corresponding optimization techniques. A standardized set of evaluation metrics and datasets is also presented to facilitate consistent and fair comparisons across different models and techniques. By offering a comprehensive overview of the current sota and identifying open research avenues, this survey serves as a foundational reference for researchers and practitioners, aiding them in developing more sustainable and efficient LLMs in a rapidly evolving landscape.