mlsanityData to 14 Oct 2026

Exploring Extreme Quantization in Spiking Language Models

arXiv 2405.02543 ↗First posted 4 May 2024cs.NEcited by 4

Not on the map. Why?

Abstract (arXiv)

Despite the growing prevalence of large language model (LLM) architectures, a crucial concern persists regarding their energy and power consumption, which still lags far behind the remarkable energy efficiency of the human brain. Recent strides in spiking language models (LM) and transformer architectures aim to address this concern by harnessing the spiking activity of biological neurons to enhance energy/power efficiency. Doubling down on the principles of model quantization and energy efficiency, this paper proposes the development of a novel binary/ternary (1/1.58-bit) spiking LM architecture. Achieving scalability comparable to a deep spiking LM architecture is facilitated by an efficient knowledge distillation technique, wherein knowledge from a non-spiking full-precision "teacher" model is transferred to an extremely weight quantized spiking "student" LM. Our proposed model represents a significant advancement as the first-of-its-kind 1/1.58-bit spiking LM, and its performance is rigorously evaluated on multiple text classification tasks of the GLUE benchmark.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 19 of its 21 references that are in the map. First posted on arXiv 4 May 2024.

Builds on

19 of 21 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2019Deep Equilibrium Modelscited by 615
  6. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  7. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 259
  8. 2018Going Deeper in Spiking Neural Networks: VGG and Residual Architecturescited by 245
  9. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  10. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 88
  11. 2022BiBERT: Accurate Fully Binarized BERTcited by 73
  12. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 56
7 more
  1. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 52
  2. 2023SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiationcited by 48
  3. 2021Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium Statecited by 40
  4. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 39
  5. 2020Exploring the Connection Between Binary and Spiking Neural Networkscited by 38
  6. 2022Post-training Quantization for Neural Networks with Provable Guaranteescited by 33
  7. 2022Sequence Learning Using Equilibrium Propagationcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?cited by 4
  2. 2025Spatio-Temporal Pruning for Compressed Spiking Large Language Modelscited by 2
  3. 2025Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigationcited by 2
  4. 2024When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mamba
  2. 2024SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking
  3. 2026Spike-driven Large Language Model
  4. 2024Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model
  5. 2026QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models
  6. 2025SpikingBrain: Spiking Brain-inspired Large Models
  7. 2026BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation
  8. 2025Memory-Free and Parallel Computation for Quantized Spiking Neural Networks