mlsanityData to 7 Oct 2026

Scalable MatMul-free Language Modeling

arXiv 2406.02528 ↗First posted 4 Jun 2024cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have fundamentally altered how we approach scaling in machine learning. However, these models pose substantial computational and memory challenges, primarily due to the reliance on matrix multiplication (MatMul) within their attention and feed-forward (FFN) layers. We demonstrate that MatMul operations can be eliminated from LLMs while maintaining strong performance, even at billion-parameter scales. Our MatMul-free models, tested on models up to 2.7B parameters, are comparable to state-of-the-art pre-trained Transformers, and the performance gap narrows as model size increases. Our approach yields significant memory savings: a GPU-efficient implementation reduces memory consumption by up to 61% during training and over 10x during inference. When adapted for a multi-chip neuromorphic system, the model leverages asynchronous processing to achieve 4x higher throughput with 10x less energy than edge GPUs.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 52 of its 70 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

52 of 70 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  3. 2020GLU Variants Improve Transformercited by 1,724
  4. 2019Root Mean Square Layer Normalizationcited by 1,425
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  7. 2016Language Modeling with Gated Convolutional Networkscited by 688
  8. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  9. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  10. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  11. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  12. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  13. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
  14. 2021Training Spiking Neural Networks Using Lessons From Deep Learningcited by 251
  15. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  16. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  17. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  18. 2020The Hardware Lotterycited by 160
  19. 2023Spike-driven Transformercited by 141
  20. 2021An Attention Free Transformercited by 111
  21. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  22. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
  23. 2021Efficient Neuromorphic Signal Processing with Loihi 2cited by 90
  24. 2019AdderNet: Do We Really Need Multiplications in Deep Learning?cited by 82
  25. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 82
  26. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  27. 2022BiBERT: Accurate Fully Binarized BERTcited by 70
  28. 2019Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answeringcited by 58
  29. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 51
  30. 2023SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiationcited by 47
  31. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
  32. 2023Efficient Video and Audio processing with Loihi 2cited by 28
  33. 2024Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticketcited by 28
  34. 2023BinaryViT: Pushing Binary Vision Transformers Towards Convolutional Modelscited by 17
  35. 2023SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillationcited by 15
  36. 2023Binarized Neural Machine Translationcited by 12
  37. 2023Binary and Ternary Natural Language Generationcited by 7
  38. 2023Multiplication-Free Transformer Training via Piecewise Affine Operationscited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 9; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024Scaling Laws for Precisioncited by 92
  2. 2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 42
  3. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 34
  4. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  5. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 27
  6. 2024Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustnesscited by 22
  7. 2025Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2cited by 6
  8. 2025Compute-in-Memory Implementation of State Space Models for Event Sequence Processingcited by 6
  9. 2024Accurate Compression of Text-to-Image Diffusion Models via Vector Quantizationcited by 3
  10. 2024Unlocking the Theory Behind Scaling 1-Bit Neural Networkscited by 3
  11. 2024Unsupervised Training of Diffusion Models for Feasible Solution Generation in Neural Combinatorial Optimizationcited by 2
  12. 2025Changing Base Without Losing Pace: A GPU-Efficient Alternative to MatMul in DNNscited by 2
9 more
  1. 2024Character-level Tokenizations as Powerful Inductive Biases for RNA Foundational Modelscited by 1
  2. 2025PROM: Prioritize Reduction of Multiplications Over Lower Bit-Widths for Efficient CNNscited by 1
  3. 2025MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Modelscited by 1
  4. 2025All-optical temporal integration mediated by subwavelength heat antennascited by 0
  5. 2025Reservoir Computing inspired Matrix Multiplication-free Language Modelcited by 0
  6. 2026Sparse PPMI Graph Averaging for Random Indexing Embeddingscited by 0
  7. 2026Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardwarecited by 0
  8. 2026Length-Independent State Tracking Under a Parallel Scancited by 0
  9. 2026Change the Product, Keep the Parameters: Associative Algebra Layers for Transformerscited by 0