# End-to-End Transformer Acceleration Through Processing-in-Memory Architectures

arXiv: https://arxiv.org/abs/2601.14260 · HTML: https://mlsanity.com/paper/2601.14260/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 21 Nov 2025.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [A3: Accelerating Attention Mechanisms in Neural Networks with Approximation](https://mlsanity.com/paper/2002.10941/) (2020)
- [Efficient Nearest Neighbor Language Models](https://mlsanity.com/paper/2109.04212/) (2021)
- [ThinK: Thinner Key Cache by Query-Driven Pruning](https://mlsanity.com/paper/2407.21018/) (2024)
- [FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs](https://mlsanity.com/paper/2401.03868/) (2024)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [Compute Or Load KV Cache? Why Not Both?](https://mlsanity.com/paper/2410.03065/) (2024)
- [GraphR: Accelerating Graph Processing Using ReRAM](https://mlsanity.com/paper/1708.06248/) (2017)
- [Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2406.08413/) (2024)
- [Multi-Objective Optimization of ReRAM Crossbars for Robust DNN Inferencing under Stochastic Noise](https://mlsanity.com/paper/2109.05437/) (2021)
- [In-Memory Nearest Neighbor Search with FeFET Multi-Bit Content-Addressable Memories](https://mlsanity.com/paper/2011.07095/) (2020)
- [Block-Wise Mixed-Precision Quantization: Enabling High Efficiency for Practical ReRAM-based DNN Accelerators](https://mlsanity.com/paper/2310.12182/) (2023)
- [AttentionLego: An Open-Source Building Block For Spatially-Scalable Large Language Model Accelerator With Processing-In-Memory Technology](https://mlsanity.com/paper/2401.11459/) (2024)
- [Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration](https://mlsanity.com/paper/2505.17787/) (2025)
- [Optimizing and Exploring System Performance in Compact Processing-in-Memory-based Chips](https://mlsanity.com/paper/2502.21259/) (2025)
- [AutoRAC: Automated Processing-in-Memory Accelerator Design for Recommender Systems](https://mlsanity.com/paper/2505.10748/) (2025)

## Built on by (most cited first, 1 of 1)

- [An RRAM-based Hardware Implementation of a Radial Basis Function Neuron for Edge Classifiers](https://mlsanity.com/paper/2606.14739/) (2026)

## Abstract (arXiv)

Transformers have become central to natural language processing and large language models, but their deployment at scale faces three major challenges. First, the attention mechanism requires massive matrix multiplications and frequent movement of intermediate results between memory and compute units, leading to high latency and energy costs. Second, in long-context inference, the key-value cache (KV cache) can grow unpredictably and even surpass the model's weight size, creating severe memory and bandwidth bottlenecks. Third, the quadratic complexity of attention with respect to sequence length amplifies both data movement and compute overhead, making large-scale inference inefficient. To address these issues, this work introduces processing-in-memory solutions that restructure attention and feed-forward computation to minimize off-chip data transfers, dynamically compress and prune the KV cache to manage memory growth, and reinterpret attention as an associative memory operation to reduce complexity and hardware footprint. Moreover, we evaluate our processing-in-memory design against state-of-the-art accelerators and general-purpose GPUs, demonstrating significant improvements in energy efficiency and latency. Together, these approaches address computation overhead, memory scalability, and attention complexity, further enabling efficient, end-to-end acceleration of Transformer models.