# X-Former: In-Memory Acceleration of Transformers

arXiv: https://arxiv.org/abs/2303.07470 · HTML: https://mlsanity.com/paper/2303.07470/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference](https://mlsanity.com/paper/2005.03842/) (2020)
- [A3: Accelerating Attention Mechanisms in Neural Networks with Approximation](https://mlsanity.com/paper/2002.10941/) (2020)
- [EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference](https://mlsanity.com/paper/2011.14203/) (2020)
- [PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference](https://mlsanity.com/paper/1901.10351/) (2019)
- [Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers](https://mlsanity.com/paper/2103.09301/) (2021)
- [Towards Fully 8-bit Integer Inference for the Transformer Model](https://mlsanity.com/paper/2009.08034/) (2020)
- [PANTHER: A Programmable Architecture for Neural Network Training Harnessing Energy-efficient ReRAM](https://mlsanity.com/paper/1912.11516/) (2019)
- [PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination](https://mlsanity.com/paper/2001.08950/) (2020)
- [TxSim:Modeling Training of Deep Neural Networks on Resistive Crossbar Systems](https://mlsanity.com/paper/2002.11151/) (2020)

## Built on by (most cited first, 25 of 25)

- [Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving](https://mlsanity.com/paper/2407.00079/) (2024)
- [A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models](https://mlsanity.com/paper/2410.07265/) (2024)
- [A Survey on Hardware Accelerators for Large Language Models](https://mlsanity.com/paper/2401.09890/) (2024)
- [Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2406.08413/) (2024)
- [Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation](https://mlsanity.com/paper/2405.01814/) (2024)
- [Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models](https://mlsanity.com/paper/2409.19315/) (2024)
- [Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators](https://mlsanity.com/paper/2505.14314/) (2025)
- [ClipFormer: Key-Value Clipping of Transformers on Memristive Crossbars for Write Noise Mitigation](https://mlsanity.com/paper/2402.02586/) (2024)
- [Hardware Acceleration of LLMs: A comprehensive survey and comparison](https://mlsanity.com/paper/2409.03384/) (2024)
- [TReX- Reusing Vision Transformer's Attention for Efficient Xbar-based Computing](https://mlsanity.com/paper/2408.12742/) (2024)
- [Atleus: Accelerating Transformers on the Edge Enabled by 3D Heterogeneous Manycore Architectures](https://mlsanity.com/paper/2501.09588/) (2025)
- [HaLoRA: Hardware-aware Low-Rank Adaptation for Large Language Models Based on Hybrid Compute-in-Memory Architecture](https://mlsanity.com/paper/2502.19747/) (2025)
- [FLASH-D: FlashAttention with Hidden Softmax Division](https://mlsanity.com/paper/2505.14201/) (2025)
- [HeTraX: Energy Efficient 3D Heterogeneous Manycore Architecture for Transformer Acceleration](https://mlsanity.com/paper/2408.03397/) (2024)
- [PIM-LLM: A High-Throughput Hybrid PIM Architecture for 1-bit LLMs](https://mlsanity.com/paper/2504.01994/) (2025)
- [AnalogNAS-Bench: A NAS Benchmark for Analog In-Memory Computing](https://mlsanity.com/paper/2506.18495/) (2025)
- [Comparative Evaluation of Memory Technologies for Synaptic Crossbar Arrays- Part 2: Design Knobs and DNN Accuracy Trends](https://mlsanity.com/paper/2408.05857/) (2024)
- [MICSim: A Modular Simulator for Mixed-signal Compute-in-Memory based AI Accelerator](https://mlsanity.com/paper/2409.14838/) (2024)
- [FLARE: FP-Less PTQ and Low-ENOB ADC Based AMS-PiM for Error-Resilient, Fast, and Efficient Transformer Acceleration](https://mlsanity.com/paper/2411.14733/) (2024)
- [FAR: Function-preserving Attention Replacement for IMC-friendly Inference](https://mlsanity.com/paper/2505.21535/) (2025)
- [HALO: Memory-Centric Heterogeneous Accelerator with 2.5D Integration for Low-Batch LLM Inference](https://mlsanity.com/paper/2510.02675/) (2025)
- [Vectorized FlashAttention with Low-cost Exponential Computation in RISC-V Vector Processors](https://mlsanity.com/paper/2510.06834/) (2025)
- [EdgeCIM: A Hardware-Software Co-Design for CIM-Based Acceleration of Small Language Models](https://mlsanity.com/paper/2604.11512/) (2026)
- [ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts](https://mlsanity.com/paper/2607.17074/) (2026)
- [RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory](https://mlsanity.com/paper/2607.21731/) (2026)

## Abstract (arXiv)

Transformers have achieved great success in a wide variety of natural language processing (NLP) tasks due to the attention mechanism, which assigns an importance score for every word relative to other words in a sequence. However, these models are very large, often reaching hundreds of billions of parameters, and therefore require a large number of DRAM accesses. Hence, traditional deep neural network (DNN) accelerators such as GPUs and TPUs face limitations in processing Transformers efficiently. In-memory accelerators based on non-volatile memory promise to be an effective solution to this challenge, since they provide high storage density while performing massively parallel matrix vector multiplications within memory arrays. However, attention score computations, which are frequently used in Transformers (unlike CNNs and RNNs), require matrix vector multiplications (MVM) where both operands change dynamically for each input. As a result, conventional NVM-based accelerators incur high write latency and write energy when used for Transformers, and further suffer from the low endurance of most NVM technologies. To address these challenges, we present X-Former, a hybrid in-memory hardware accelerator that consists of both NVM and CMOS processing elements to execute transformer workloads efficiently. To improve the hardware utilization of X-Former, we also propose a sequence blocking dataflow, which overlaps the computations of the two processing elements and reduces execution time. Across several benchmarks, we show that X-Former achieves upto 85x and 7.5x improvements in latency and energy over a NVIDIA GeForce GTX 1060 GPU and upto 10.7x and 4.6x improvements in latency and energy over a state-of-the-art in-memory NVM accelerator.