# Matterhorn: Masked Time-to-First-Spike Encoding by Reassigning the Silent State for Sparse and Energy-Efficient Spiking Transformers

arXiv: https://arxiv.org/abs/2601.22876 · HTML: https://mlsanity.com/paper/2601.22876/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [BiT: Robustly Binarized Multi-distilled Transformer](https://mlsanity.com/paper/2205.13016/) (2022)
- [SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation](https://mlsanity.com/paper/2308.10873/) (2023)
- [Neural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networks](https://mlsanity.com/paper/1805.03718/) (2018)
- [SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms](https://mlsanity.com/paper/2406.03287/) (2024)
- [SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking](https://mlsanity.com/paper/2407.04752/) (2024)
- [Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model](https://mlsanity.com/paper/2409.15298/) (2024)
- [A Time-to-first-spike Coding and Conversion Aware Training for Energy-Efficient Deep Spiking Neural Network Processor Design](https://mlsanity.com/paper/2208.04494/) (2022)
- [Otters: An Energy-Efficient SpikingTransformer via Optical Time-to-First-Spike Encoding](https://mlsanity.com/paper/2509.18968/) (2025)
- [Sparse-firing regularization methods for spiking neural networks with time-to-first spike coding](https://mlsanity.com/paper/2307.13007/) (2023)
- [Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigation](https://mlsanity.com/paper/2510.19498/) (2025)
- [SpikeX: Exploring Accelerator Architecture and Network-Hardware Co-Optimization for Sparse Spiking Neural Networks](https://mlsanity.com/paper/2505.12292/) (2025)

## Abstract (arXiv)

Spiking neural networks (SNNs) promise energy-efficient inference for large language models (LLMs), yet most reported savings rely on compute-operation counts that overlook data movement. Energy characterization of representative spiking transformers on a commercial 22-nm process shows that accumulation contributes less than 3% of total energy, while spike-triggered inter-core transfers and weight reads dominate the cost. This makes time-to-first-spike (TTFS) encoding a natural choice, as it limits each neuron to at most one spike. However, standard TTFS maps the silent state, an all-zero spike train that transmits no events, to the rarely occurring smallest value, while the most common activations still spike. This raises a simple question: why reserve the only cost-free codeword for a rare value? This choice inverts a basic principle of energy-aware coding, under which the zero-event codeword should represent the most common value, rather than a rare extreme. Thus, we introduce masked time-to-first-spike encoding (M-TTFS), which uses a temporal mask to reassign the silent state to the most common activation value, and a dead-zone extension that trades a controlled amount of information for greater sparsity. Built on M-TTFS with dead-zone radius k=1, our spiking transformer Matterhorn reaches an overall spike rate of 1.64% on GLUE at an average score of 84.64, exceeding the best prior spiking transformer by 1.42 percentage points while consuming 67% less energy, with consistent gains on spiking LLaMA models from 7B to 70B parameters. Together, these results show that under hardware-faithful accounting, the energy advantage of SNNs is not a given: it is earned by encodings that align spikes with the data distribution.