# Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference

arXiv: https://arxiv.org/abs/2406.08413 · HTML: https://mlsanity.com/paper/2406.08413/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 58 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs](https://mlsanity.com/paper/2401.03868/) (2024)
- [PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference](https://mlsanity.com/paper/1901.10351/) (2019)
- [Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators](https://mlsanity.com/paper/2302.08469/) (2023)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [X-Former: In-Memory Acceleration of Transformers](https://mlsanity.com/paper/2303.07470/) (2023)
- [Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models](https://mlsanity.com/paper/2205.09646/) (2022)
- [A Hardware Evaluation Framework for Large Language Model Inference](https://mlsanity.com/paper/2312.03134/) (2023)
- [PIM-GPT: A Hybrid Process-in-Memory Accelerator for Autoregressive Transformers](https://mlsanity.com/paper/2310.09385/) (2023)
- [A Heterogeneous Chiplet Architecture for Accelerating End-to-End Transformer Models](https://mlsanity.com/paper/2312.11750/) (2023)
- [Memory-Centric Computing](https://mlsanity.com/paper/2305.20000/) (2023)
- [Zero-Space Cost Fault Tolerance for Transformer-based Language Models on ReRAM](https://mlsanity.com/paper/2401.11664/) (2024)
- [Biologically Plausible Learning on Neuromorphic Hardware Architectures](https://mlsanity.com/paper/2212.14337/) (2022)

## Built on by (most cited first, 15 of 15)

- [DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively](https://mlsanity.com/paper/2509.26603/) (2025)
- [Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective](https://mlsanity.com/paper/2410.04466/) (2024)
- [Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models](https://mlsanity.com/paper/2409.19315/) (2024)
- [AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling](https://mlsanity.com/paper/2507.08567/) (2025)
- [From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefill](https://mlsanity.com/paper/2510.08055/) (2025)
- [Model-Native Computing Architecture: Envisioning Future System Architecture Through the Lens of Computer Architecture](https://mlsanity.com/paper/2606.00288/) (2026)
- [PIM-LLM: A High-Throughput Hybrid PIM Architecture for 1-bit LLMs](https://mlsanity.com/paper/2504.01994/) (2025)
- [A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent](https://mlsanity.com/paper/2509.19209/) (2025)
- [End-to-End Transformer Acceleration Through Processing-in-Memory Architectures](https://mlsanity.com/paper/2601.14260/) (2025)
- [LLM Inference in a Flash!](https://mlsanity.com/paper/2609.16161/) (2026)
- [FAR: Function-preserving Attention Replacement for IMC-friendly Inference](https://mlsanity.com/paper/2505.21535/) (2025)
- [Energy Efficient Software Hardware CoDesign for Machine Learning: From TinyML to Large Language Models](https://mlsanity.com/paper/2603.23668/) (2026)
- [A First-Principles Theory of Slow Thinking and Active Perception](https://mlsanity.com/paper/2607.08196/) (2026)
- [FastTPS: An Optimized Method for LLM Token Phase for AI accelerators](https://mlsanity.com/paper/2607.11211/) (2026)
- [Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers](https://mlsanity.com/paper/2608.21223/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have recently transformed natural language processing, enabling machines to generate human-like text and engage in meaningful conversations. This development necessitates speed, efficiency, and accessibility in LLM inference as the computational and memory requirements of these systems grow exponentially. Meanwhile, advancements in computing and memory capabilities are lagging behind, exacerbated by the discontinuation of Moore's law. With LLMs exceeding the capacity of single GPUs, they require complex, expert-level configurations for parallel processing. Memory accesses become significantly more expensive than computation, posing a challenge for efficient scaling, known as the memory wall. Here, compute-in-memory (CIM) technologies offer a promising solution for accelerating AI inference by directly performing analog computations in memory, potentially reducing latency and power consumption. By closely integrating memory and compute elements, CIM eliminates the von Neumann bottleneck, reducing data movement and improving energy efficiency. This survey paper provides an overview and analysis of transformer-based models, reviewing various CIM architectures and exploring how they can address the imminent challenges of modern AI computing systems. We discuss transformer-related operators and their hardware acceleration schemes and highlight challenges, trends, and insights in corresponding CIM designs.