# Scalable MatMul-free Language Modeling

arXiv: https://arxiv.org/abs/2406.02528 · HTML: https://mlsanity.com/paper/2406.02528/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 70 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://mlsanity.com/paper/2402.19427/) (2024)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [BinaryBERT: Pushing the Limit of BERT Quantization](https://mlsanity.com/paper/2012.15701/) (2020)
- [The Hardware Lottery](https://mlsanity.com/paper/2009.06489/) (2020)

## Built on by (most cited first, 25 of 25)

- [Scaling Laws for Precision](https://mlsanity.com/paper/2411.04330/) (2024)
- [BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulation](https://mlsanity.com/paper/2506.07530/) (2025)
- [RandLoRA: Full-rank parameter-efficient fine-tuning of large models](https://mlsanity.com/paper/2502.00987/) (2025)
- [ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization](https://mlsanity.com/paper/2406.05981/) (2024)
- [u-μP: The Unit-Scaled Maximal Update Parametrization](https://mlsanity.com/paper/2407.17465/) (2024)
- [Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustness](https://mlsanity.com/paper/2408.04585/) (2024)
- [Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2](https://mlsanity.com/paper/2503.18002/) (2025)
- [TerEffic: Highly Efficient Ternary LLM Inference on FPGA](https://mlsanity.com/paper/2502.16473/) (2025)
- [Compute-in-Memory Implementation of State Space Models for Event Sequence Processing](https://mlsanity.com/paper/2511.13912/) (2025)
- [Unlocking the Theory Behind Scaling 1-Bit Neural Networks](https://mlsanity.com/paper/2411.01663/) (2024)
- [Accurate Compression of Text-to-Image Diffusion Models via Vector Quantization](https://mlsanity.com/paper/2409.00492/) (2024)
- [Unsupervised Training of Diffusion Models for Feasible Solution Generation in Neural Combinatorial Optimization](https://mlsanity.com/paper/2411.00003/) (2024)
- [Changing Base Without Losing Pace: A GPU-Efficient Alternative to MatMul in DNNs](https://mlsanity.com/paper/2503.12211/) (2025)
- [PROM: Prioritize Reduction of Multiplications Over Lower Bit-Widths for Efficient CNNs](https://mlsanity.com/paper/2505.03254/) (2025)
- [Character-level Tokenizations as Powerful Inductive Biases for RNA Foundational Models](https://mlsanity.com/paper/2411.11808/) (2024)
- [MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Models](https://mlsanity.com/paper/2506.14435/) (2025)
- [All-optical temporal integration mediated by subwavelength heat antennas](https://mlsanity.com/paper/2505.04405/) (2025)
- [Reservoir Computing inspired Matrix Multiplication-free Language Model](https://mlsanity.com/paper/2512.23145/) (2025)
- [BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design](https://mlsanity.com/paper/2604.03957/) (2026)
- [Spike-driven Large Language Model](https://mlsanity.com/paper/2604.16475/) (2026)
- [Sparse PPMI Graph Averaging for Random Indexing Embeddings](https://mlsanity.com/paper/2608.05724/) (2026)
- [Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware](https://mlsanity.com/paper/2608.30439/) (2026)
- [Length-Independent State Tracking Under a Parallel Scan](https://mlsanity.com/paper/2609.32447/) (2026)
- [Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers](https://mlsanity.com/paper/2609.32814/) (2026)
- [ADPTNet: Adaptive with Prescriptive Timescales Non-Linear SSM for Sequence Modelling](https://mlsanity.com/paper/2609.34034/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have fundamentally altered how we approach scaling in machine learning. However, these models pose substantial computational and memory challenges, primarily due to the reliance on matrix multiplication (MatMul) within their attention and feed-forward (FFN) layers. We demonstrate that MatMul operations can be eliminated from LLMs while maintaining strong performance, even at billion-parameter scales. Our MatMul-free models, tested on models up to 2.7B parameters, are comparable to state-of-the-art pre-trained Transformers, and the performance gap narrows as model size increases. Our approach yields significant memory savings: a GPU-efficient implementation reduces memory consumption by up to 61% during training and over 10x during inference. When adapted for a multi-chip neuromorphic system, the model leverages asynchronous processing to achieve 4x higher throughput with 10x less energy than edge GPUs.