# Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware

arXiv: https://arxiv.org/abs/2608.30439 · HTML: https://mlsanity.com/paper/2608.30439/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 30 of 34)

- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://mlsanity.com/paper/2310.17157/) (2023)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks](https://mlsanity.com/paper/1708.04485/) (2017)
- [Neural Pruning via Growing Regularization](https://mlsanity.com/paper/2012.09243/) (2020)
- [ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models](https://mlsanity.com/paper/2310.04564/) (2023)
- [ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMs](https://mlsanity.com/paper/2402.03804/) (2024)
- [Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering](https://mlsanity.com/paper/1911.07176/) (2019)
- [Training-Free Activation Sparsity in Large Language Models](https://mlsanity.com/paper/2408.14690/) (2024)
- [A Systematic Analysis of Hybrid Linear Attention](https://mlsanity.com/paper/2507.06457/) (2025)
- [Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters](https://mlsanity.com/paper/2406.05955/) (2024)
- [Scalable MatMul-free Language Modeling](https://mlsanity.com/paper/2406.02528/) (2024)
- [Q-Sparse: All Large Language Models can be Fully Sparsely-Activated](https://mlsanity.com/paper/2407.10969/) (2024)
- [One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models](https://mlsanity.com/paper/2310.09499/) (2023)
- [Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models](https://mlsanity.com/paper/2503.22879/) (2025)
- [Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2](https://mlsanity.com/paper/2503.18002/) (2025)
- [On Pruning State-Space LLMs](https://mlsanity.com/paper/2502.18886/) (2025)
- [The Resurrection of the ReLU](https://mlsanity.com/paper/2505.22074/) (2025)

## Abstract (arXiv)

Inference with transformer-based large language models (LLMs) is often limited by the memory-bound KV cache and quadratic attention cost. State-space models (SSMs) mitigate this through linear attention and fixed-size recurrent states, but their large dense linear projections remain computationally expensive even after quantization. We introduce a method that induces sparse neural activity in heavily quantized linear-attention models with minimal performance loss. Activations below a per-projection trainable threshold (± Delta) are nullified while preserving crucial outliers, achieving comparable performance to dense models with up to 4× fewer effective arithmetic operations. Targeting a multi-core, multi-chip neuromorphic platform, where event-driven execution converts unstructured sparsity into throughput at both the compute and communication levels, a capability GPU architectures fundamentally lack, we project up to 37× higher throughput and 16× lower power versus edge GPU inference of a comparable transformer-based model, and up to 5.4× improvements over the non-sparsified baseline. These results position sparse, quantized linear-attention models as a natural fit for deploying LLMs on event-driven multi-core platforms.