mlsanityData to 7 Oct 2026

Reservoir Computing inspired Matrix Multiplication-free Language Model

arXiv 2512.23145 ↗First posted 29 Dec 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have achieved state-of-the-art performance in natural language processing; however, their high computational cost remains a major bottleneck. In this study, we target computational efficiency by focusing on a matrix multiplication free language model (MatMul-free LM) and further reducing the training cost through an architecture inspired by reservoir computing. Specifically, we partially fix and share the weights of selected layers in the MatMul-free LM and insert reservoir layers to obtain rich dynamic representations without additional training overhead. Additionally, several operations are combined to reduce memory accesses. Experimental results show that the proposed architecture reduces the number of parameters by up to 19%, training time by 9.9%, and inference time by 8.0%, while maintaining comparable performance to the baseline model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 60 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

41 of 60 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
29 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  9. 2016Language Modeling with Gated Convolutional Networkscited by 688
  10. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  11. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  12. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 180
  13. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  14. 2019Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamicscited by 132
  15. 2023Large Language Models for Robotics: A Surveycited by 132
  16. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  17. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  18. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 100
  19. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 40
  20. 2024Large Language Models for Medicine: A Surveycited by 38
  21. 2023Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languagescited by 36
  22. 2023Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Modelscited by 35
  23. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  24. 2024Scalable MatMul-free Language Modelingcited by 21
  25. 2024MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Modelscited by 21
  26. 2020Reservoir Transformerscited by 18
  27. 2024Characterizing the Accuracy -- Efficiency Trade-off of Low-rank Decomposition in Language Modelscited by 4
  28. 2024OPAL: Outlier-Preserved Microscaling Quantization Accelerator for Generative Large Language Modelscited by 3
  29. 2024Unified Understanding of Environment, Task, and Human for Human-Robot Interaction in Real-World Environmentscited by 2

Built on by

Not cited yet.