mlsanityData to 7 Oct 2026

A Survey of Resource-efficient LLM and Multimodal Foundation Models

arXiv 2401.08092 ↗First posted 16 Jan 2024cs.LG cs.AI cs.DCcited by 84

Not on the map. Why?

Abstract (arXiv)

Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine learning lifecycle, from training to deployment. However, the substantial advancements in versatility and performance these models offer come at a significant cost in terms of hardware resources. To support the growth of these large models in a scalable and environmentally sustainable way, there has been a considerable focus on developing resource-efficient strategies. This survey delves into the critical importance of such research, examining both algorithmic and systemic aspects. It offers a comprehensive analysis and valuable insights gleaned from existing literature, encompassing a broad array of topics from cutting-edge model architectures and training/serving algorithms to practical system designs and implementations. The goal of this survey is to provide an overarching understanding of how current approaches are tackling the resource challenges posed by large foundation models and to potentially inspire future breakthroughs in this field.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 299 of its 460 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

299 of 460 references.

References · 299

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2022Training language models to follow instructions with human feedbackcited by 15,712
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  11. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  12. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
38 more
  1. 2020Denoising Diffusion Implicit Modelscited by 9,655
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  13. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  14. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  15. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  16. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  17. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  18. 2013Auto-Encoding Variational Bayescited by 3,980
  19. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  20. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  21. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  22. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  23. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  24. 2020Longformer: The Long-Document Transformercited by 3,362
  25. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  26. 2023Mistral 7Bcited by 3,065
  27. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  28. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  29. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  30. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  31. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  32. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  33. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  34. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  35. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  36. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  37. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  38. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961

50 most cited of 299 All 299 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 43; 2025: 26; 2026: 15.

84 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  4. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  5. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  6. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  7. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  8. 2024Fast On-device LLM Inference with NPUscited by 58
  9. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  10. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  11. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  12. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 35
38 more
  1. 2025Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audiocited by 31
  2. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
  3. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  4. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  5. 2025Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agentscited by 24
  6. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  7. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  8. 2024Federated Large Language Models: Current Progress and Future Directionscited by 22
  9. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipelinecited by 20
  10. 2024A Unified Debiasing Approach for Vision-Language Models across Modalities and Taskscited by 18
  11. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  12. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  13. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  14. 2024Text-centric Alignment for Multi-Modality Learningcited by 12
  15. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 11
  16. 2024Leveraging Large Language Models for Integrated Satellite-Aerial-Terrestrial Networks: Recent Advances and Future Directionscited by 10
  17. 2024GenBFA: An Evolutionary Optimization Approach to Bit-Flip Attacks on LLMscited by 10
  18. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  19. 2024Transformer Explainer: Learning LLM Transformers with Interactive Visual Explanation and Experimentationcited by 9
  20. 2025Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospectscited by 9
  21. 2025Thinking Preference Optimizationcited by 8
  22. 2025HuggingGraph: Understanding the Supply Chain of LLM Ecosystemcited by 8
  23. 2024Mamba as a motion encoder for robotic imitation learningcited by 7
  24. 2024PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-trainingcited by 7
  25. 2024Deep Autoregressive Models as Causal Inference Enginescited by 6
  26. 2025Leveraging the true depth of LLMscited by 6
  27. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  28. 2025Diffusion Augmented Retrieval: A Training-Free Approach to Interactive Text-to-Image Retrievalcited by 5
  29. 2025RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Modelscited by 5
  30. 2025Collaborative Speculative Inference for Efficient LLM Inference Servingcited by 5
  31. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  32. 2026When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoningcited by 5
  33. 2026BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimizationcited by 5
  34. 2024VernaCopter: Disambiguated Natural-Language-Driven Robot via Formal Specificationscited by 4
  35. 2024LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Modelscited by 4
  36. 2024Dynamic Vocabulary Pruning in Early-Exit LLMscited by 4
  37. 2024DroidCall: A Dataset for LLM-powered Android Intent Invocationcited by 4
  38. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4

50 most cited of 84 All 84 (JSON)