# SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation

arXiv: https://arxiv.org/abs/2308.10873 · HTML: https://mlsanity.com/paper/2308.10873/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2023.

## Built on by (most cited first, 30 of 48)

- [SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking](https://mlsanity.com/paper/2407.04752/) (2024)
- [Scalable MatMul-free Language Modeling](https://mlsanity.com/paper/2406.02528/) (2024)
- [SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN](https://mlsanity.com/paper/2406.03470/) (2024)
- [SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation](https://mlsanity.com/paper/2308.15122/) (2023)
- [Autonomous Driving with Spiking Neural Networks](https://mlsanity.com/paper/2405.19687/) (2024)
- [Reconsidering the Energy Efficiency of Spiking Neural Networks Inference from Analytical Perspectives](https://mlsanity.com/paper/2409.08290/) (2024)
- [Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networks](https://mlsanity.com/paper/2406.07862/) (2024)
- [Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directions](https://mlsanity.com/paper/2409.02111/) (2024)
- [Language Modeling on a SpiNNaker 2 Neuromorphic Chip](https://mlsanity.com/paper/2312.09084/) (2023)
- [Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model](https://mlsanity.com/paper/2409.15298/) (2024)
- [FTBC: Forward Temporal Bias Correction for Optimizing ANN-SNN Conversion](https://mlsanity.com/paper/2403.18388/) (2024)
- [Phi: Leveraging Pattern-based Hierarchical Sparsity for High-Efficiency Spiking Neural Networks](https://mlsanity.com/paper/2505.10909/) (2025)
- [Otters: An Energy-Efficient SpikingTransformer via Optical Time-to-First-Spike Encoding](https://mlsanity.com/paper/2509.18968/) (2025)
- [SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Network](https://mlsanity.com/paper/2310.06488/) (2023)
- [Exploring Extreme Quantization in Spiking Language Models](https://mlsanity.com/paper/2405.02543/) (2024)
- [SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mamba](https://mlsanity.com/paper/2510.04595/) (2025)
- [QSLM: A Performance- and Memory-aware Quantization Framework with Tiered Search Strategy for Spike-driven Language Models](https://mlsanity.com/paper/2601.00679/) (2026)
- [Benchmarking Spiking Neural Network Learning Methods with Varying Locality](https://mlsanity.com/paper/2402.01782/) (2024)
- [Stochastic Spiking Attention: Accelerating Attention with Stochastic Computing in Spiking Networks](https://mlsanity.com/paper/2402.09109/) (2024)
- [Temporal Reversal Regularization for Spiking Neural Networks: Hybrid Spatio-Temporal Invariance for Generalization](https://mlsanity.com/paper/2408.09108/) (2024)
- [FAS: Fast ANN-SNN Conversion for Spiking Large Language Models](https://mlsanity.com/paper/2502.04405/) (2025)
- [Edge Intelligence with Spiking Neural Networks](https://mlsanity.com/paper/2507.14069/) (2025)
- [Spatio-Temporal Pruning for Compressed Spiking Large Language Models](https://mlsanity.com/paper/2508.20122/) (2025)
- [Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers](https://mlsanity.com/paper/2510.07924/) (2025)
- [Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigation](https://mlsanity.com/paper/2510.19498/) (2025)
- [Modern Neuromorphic AI: From Intra-Token to Inter-Token Processing](https://mlsanity.com/paper/2601.00245/) (2026)
- [On Membership Inference Attacks in Knowledge Distillation](https://mlsanity.com/paper/2505.11837/) (2025)
- [SVL: Empowering Spiking Neural Networks for Efficient 3D Open-World Understanding](https://mlsanity.com/paper/2505.17674/) (2025)
- [IML-Spikeformer: Input-aware Multi-Level Spiking Transformer for Speech Processing](https://mlsanity.com/paper/2507.07396/) (2025)
- [Rethinking LLM-Based RTL Code Optimization Via Timing Logic Metamorphosis](https://mlsanity.com/paper/2507.16808/) (2025)

## Abstract (arXiv)

Large language Models (LLMs), though growing exceedingly powerful, comprises of orders of magnitude less neurons and synapses than the human brain. However, it requires significantly more power/energy to operate. In this work, we propose a novel bio-inspired spiking language model (LM) which aims to reduce the computational cost of conventional LMs by drawing motivation from the synaptic information flow in the brain. In this paper, we demonstrate a framework that leverages the average spiking rate of neurons at equilibrium to train a neuromorphic spiking LM using implicit differentiation technique, thereby overcoming the non-differentiability problem of spiking neural network (SNN) based algorithms without using any type of surrogate gradient. The steady-state convergence of the spiking neurons also allows us to design a spiking attention mechanism, which is critical in developing a scalable spiking LM. Moreover, the convergence of average spiking rate of neurons at equilibrium is utilized to develop a novel ANN-SNN knowledge distillation based technique wherein we use a pre-trained BERT model as "teacher" to train our "student" spiking architecture. While the primary architecture proposed in this paper is motivated by BERT, the technique can be potentially extended to different kinds of LLMs. Our work is the first one to demonstrate the performance of an operational spiking LM architecture on multiple different tasks in the GLUE benchmark.