mlsanityData to 7 Oct 2026

Designing Large Foundation Models for Efficient Training and Inference: A Survey

arXiv 2409.01990 ↗First posted 3 Sep 2024cs.DC cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper focuses on modern efficient training and inference technologies on foundation models and illustrates them from two perspectives: model and system design. Model and System Design optimize LLM training and inference from different aspects to save computational resources, making LLMs more efficient, affordable, and more accessible. The paper list repository is available at https://github.com/NoakLiu/Efficient-Foundation-Models-Survey.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 121 of its 187 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

121 of 187 references.

References · 121

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  13. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  14. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  15. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  16. 2017Mixed Precision Trainingcited by 1,135
  17. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  18. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  19. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  20. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  21. 2022Token Merging: Your ViT But Fastercited by 995
  22. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  23. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  24. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  25. 2016Sequence-Level Knowledge Distillationcited by 761
  26. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  27. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  28. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  29. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  30. 2019Learned Step Size Quantizationcited by 604
  31. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  32. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  33. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  34. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  35. 2019The State of Sparsity in Deep Neural Networkscited by 471
  36. 2021A White Paper on Neural Network Quantizationcited by 453
  37. 2021MetaICL: Learning to Learn In Contextcited by 450
  38. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408

50 most cited of 121 All 121 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 53
  2. 2025CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inferencecited by 7
  3. 2025PiKV: KV Cache Management System for Mixture of Expertscited by 5
  4. 2025Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTScited by 4
  5. 2025SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modelingcited by 3
  6. 2025Cognitive Load Traces as Symbolic and Visual Accounts of Deep Model Cognitioncited by 0
  7. 2026Safe and Adaptive Cloud Healing: Verifying LLM-Generated Recovery Plans with a Neural-Symbolic World Modelcited by 0