mlsanityData to 7 Oct 2026

SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms

arXiv 2406.03287 ↗First posted 5 Jun 2024cs.NE cs.CL cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Towards energy-efficient artificial intelligence similar to the human brain, the bio-inspired spiking neural networks (SNNs) have advantages of biological plausibility, event-driven sparsity, and binary activation. Recently, large-scale language models exhibit promising generalization capability, making it a valuable issue to explore more general spike-driven models. However, the binary spikes in existing SNNs fail to encode adequate semantic information, placing technological challenges for generalization. This work proposes the first fully spiking mechanism for general language tasks, including both discriminative and generative ones. Different from previous spikes with {0,1} levels, we propose a more general spike formulation with bi-directional, elastic amplitude, and elastic frequency encoding, while still maintaining the addition nature of SNNs. In a single time step, the spike is enhanced by direction and amplitude information; in spike frequency, a strategy to control spike firing rate is well designed. We plug this elastic bi-spiking mechanism in language modeling, named SpikeLM. It is the first time to handle general language tasks with fully spike-driven models, which achieve much higher accuracy than previously possible. SpikeLM also greatly bridges the performance gap between SNNs and ANNs in language modeling. Our code is available at https://github.com/Xingrun-Xing/SpikeLM.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 48 of its 71 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

48 of 71 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Segment Anythingcited by 8,158
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
36 more
  1. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  2. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 399
  3. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 340
  4. 2021Deep Residual Learning in Spiking Neural Networkscited by 326
  5. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  6. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 268
  7. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  8. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  9. 2019Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detectioncited by 174
  10. 2023SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligencecited by 163
  11. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  12. 2023Spike-driven Transformercited by 141
  13. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 134
  14. 2023Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networkscited by 116
  15. 2021Accurate and efficient time-domain classification with adaptive spiking recurrent neural networkscited by 113
  16. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
  17. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  18. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 82
  19. 2023Deep Directly-Trained Spiking Neural Networks for Object Detectioncited by 76
  20. 2021Brain-inspired computing: We need a master plancited by 51
  21. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 51
  22. 2023Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipelinecited by 50
  23. 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 48
  24. 2020Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networkscited by 46
  25. 2023Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependenciescited by 41
  26. 2023Ternary Spike: Learning Ternary Spikes for Spiking Neural Networkscited by 36
  27. 2023Membrane Potential Batch Normalization for Spiking Neural Networkscited by 35
  28. 2023Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorizationcited by 34
  29. 2023Direct Learning-Based Deep Spiking Neural Networks: A Reviewcited by 31
  30. 2024Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticketcited by 28
  31. 2022Spiking Neural Network Integrated Circuits: A Review of Trends and Future Directionscited by 27
  32. 2023RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networkscited by 27
  33. 2022DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantizationcited by 23
  34. 2020A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networkscited by 8
  35. 2022Towards Accurate Binary Neural Networks via Modeling Contextual Dependenciescited by 6
  36. 2023BiPFT: Binary Pre-trained Foundation Transformer with Low-rank Estimation of Binarization Residual Polynomialscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 15.

31 papers.

Sort:

By area:

  1. 2024SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spikingcited by 29
  2. 2024Reconsidering the Energy Efficiency of Spiking Neural Networks Inference from Analytical Perspectivescited by 12
  3. 2025SDTrack: A Baseline for Event-based Tracking via Spiking Neural Networkscited by 11
  4. 2024Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Modelcited by 6
  5. 2025Otters: An Energy-Efficient SpikingTransformer via Optical Time-to-First-Spike Encodingcited by 5
  6. 2025Phi: Leveraging Pattern-based Hierarchical Sparsity for High-Efficiency Spiking Neural Networkscited by 4
  7. 2026TP-Spikformer: Token Pruned Spiking Transformercited by 4
  8. 2025Improving Performance of Spike-based Deep Q-Learning using Ternary Neuronscited by 3
  9. 2025SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mambacited by 3
  10. 2026QSLM: A Performance- and Memory-aware Quantization Framework with Tiered Search Strategy for Spike-driven Language Modelscited by 3
  11. 2025Towards Efficient and Accurate Spiking Neural Networks via Adaptive Bit Allocationcited by 2
  12. 2025Edge Intelligence with Spiking Neural Networkscited by 2
19 more
  1. 2025SpikCommander: A High-performance Spiking Transformer with Multi-view Learning for Efficient Speech Command Recognitioncited by 2
  2. 2024SDP: Spiking Diffusion Policy for Robotic Manipulation with Learnable Channel-Wise Membrane Thresholdscited by 1
  3. 2025FAS: Fast ANN-SNN Conversion for Spiking Large Language Modelscited by 1
  4. 2025Decision SpikeFormer: Spike-Driven Transformer for Decision Makingcited by 1
  5. 2025IML-Spikeformer: Input-aware Multi-Level Spiking Transformer for Speech Processingcited by 1
  6. 2026BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillationcited by 1
  7. 2026A2SG:Adaptive and Asymmetric Surrogate Gradients for Training Deep Spiking Neural Networkscited by 1
  8. 2026SpikeVLA: Vision-Language-Action Models with Spiking Neural Networkscited by 1
  9. 2025Spiking Vocos: An Energy-Efficient Neural Vocodercited by 0
  10. 2026NEXUS: Bit-Exact ANN-to-SNN Equivalence via Neuromorphic Gate Circuits with Surrogate-Free Trainingcited by 0
  11. 2026Matterhorn: Masked Time-to-First-Spike Encoding by Reassigning the Silent State for Sparse and Energy-Efficient Spiking Transformerscited by 0
  12. 2026DBHN-Net: Dual-Branch Hybrid Neural Network For Low-Complexity Monaural Speech Enhancementcited by 0
  13. 2026LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learningcited by 0
  14. 2026Otters++: A Time-to-first-spike Based Energy Efficient Optical Spiking Transformercited by 0
  15. 2026SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Modelscited by 0
  16. 2026Large Language Models with At Most One Spike per Neuroncited by 0
  17. 2026SymbolicLight V2: Hybrid Neuromorphic Architecture and Sparse Execution for Low-Energy Language Inferencecited by 0
  18. 2026URCHIN: A Horizontal Spiking Language Model for Data-Constrained Pretrainingcited by 0
  19. 2026SpikeMoE: Brain-Inspired Competitive Routing for Flexible Spiking Mixture-of-Expertscited by 0