mlsanityData to 7 Oct 2026

AI and Memory Wall

arXiv 2403.14123 ↗First posted 21 Mar 2024cs.LG cs.AR cs.DCcited by 89

Not on the map. Why?

Cited by 23 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The availability of unprecedented unsupervised training data, along with neural scaling laws, has resulted in an unprecedented surge in model size and compute requirements for serving/training LLMs. However, the main performance bottleneck is increasingly shifting to memory bandwidth. Over the past 20 years, peak server hardware FLOPS has been scaling at 3.0x/2yrs, outpacing the growth of DRAM and interconnect bandwidth, which have only scaled at 1.6 and 1.4 times every 2 years, respectively. This disparity has made memory, rather than compute, the primary bottleneck in AI applications, particularly in serving. Here, we analyze encoder and decoder Transformer models and show how memory bandwidth can become the dominant bottleneck for decoder models. We argue for a redesign in model architecture, training, and deployment strategies to overcome this memory limitation.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Builds on 26 of its 46 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

26 of 46 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Mistral 7Bcited by 3,065
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  12. 2017Mixed Precision Trainingcited by 1,135
14 more
  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  4. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  5. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  6. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  7. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  8. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  9. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  10. 2022FP8 Formats for Deep Learningcited by 187
  11. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  12. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 90
  13. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  14. 2020Accelerating Recommender Systems via Hardware "scale-in"cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 30; 2026: 33.

89 papers.

Sort:

By area:

  1. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  2. 2025GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidancecited by 28
  3. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 27
  4. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  5. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  6. 2025Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuningcited by 17
  7. 2024Flex-PE: Flexible and SIMD Multi-Precision Processing Element for AI Workloadscited by 15
  8. 2025ETS: Efficient Tree Search for Inference-Time Scalingcited by 15
  9. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12
  10. 2024BitMoD: Bit-serial Mixture-of-Datatype LLM Accelerationcited by 12
  11. 2024Recent and Upcoming Developments in Randomized Numerical Linear Algebra for Machine Learningcited by 11
  12. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Controlcited by 11
38 more
  1. 2024Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Maskingcited by 10
  2. 2024TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computerscited by 9
  3. 2026HiFloat4 Format for Language Model Inferencecited by 9
  4. 2024Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learningcited by 8
  5. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inferencecited by 8
  6. 2024FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awarenesscited by 7
  7. 2024A Diagonal Structured State Space Model on Loihi 2 for Efficient Streaming Sequence Processingcited by 6
  8. 2024SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantizationcited by 6
  9. 2025Leveraging the true depth of LLMscited by 6
  10. 2025Tawa: Automatic Warp Specialization for Modern GPUs with Asynchronous Referencescited by 6
  11. 2025Fine-tuning Quantized Neural Networks with Zeroth-order Optimizationcited by 5
  12. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Searchcited by 5
  13. 2026CoreQ: Learning-Free Mismatch Correction and Successive Rounding for Quantizationcited by 5
  14. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 5
  15. 2025POLARON: Precision-aware On-device Learning and Adaptive Runtime-cONfigurable AI accelerationcited by 4
  16. 2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerializationcited by 4
  17. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limitscited by 4
  18. 2026DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reducecited by 4
  19. 2026LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Evictioncited by 4
  20. 2026AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantizationcited by 4
  21. 2024Reducing the Barriers to Entry for Foundation Model Trainingcited by 3
  22. 2024BBS: Bi-directional Bit-level Sparsity for Deep Learning Accelerationcited by 3
  23. 2025A Survey on Memory-Efficient Transformer-Based Model Training in AI for Sciencecited by 3
  24. 2025Determinant Estimation under Memory Constraints and Neural Scaling Lawscited by 3
  25. 2026CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuningcited by 3
  26. 2024PIM-AI: A Novel Architecture for High-Efficiency LLM Inferencecited by 2
  27. 2025Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scalecited by 2
  28. 2025Arbitrage: Efficient Reasoning via Advantage-Aware Speculationcited by 2
  29. 2026RAP: KV-Cache Compression via RoPE-Aligned Pruningcited by 2
  30. 2026OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantizationcited by 2
  31. 2026AURA: Action-Gated Memory for Robot Policies at Constant VRAMcited by 2
  32. 2024OPIMA: Optical Processing-In-Memory for Convolutional Neural Network Accelerationcited by 1
  33. 2024On Exact Bit-level Reversible Transformers Without Changing Architecturescited by 1
  34. 2024LightCL: Compact Continual Learning with Low Memory Footprint For Edge Devicecited by 1
  35. 2025CIMFlow: An Integrated Framework for Systematic Design and Evaluation of Digital CIM Architecturescited by 1
  36. 2025EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devicescited by 1
  37. 2025Memory Access Characterization of Large Language Models in CPU Environment and its Potential Impactscited by 1
  38. 2025Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory Systemcited by 1

50 most cited of 89 All 89 (JSON)