# Modern Neuromorphic AI: From Intra-Token to Inter-Token Processing

arXiv: https://arxiv.org/abs/2601.00245 · HTML: https://mlsanity.com/paper/2601.00245/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 64 references that are in the map.
- First posted on arXiv 1 Jan 2026.

## Builds on (most cited first, 30 of 45)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections](https://mlsanity.com/paper/2206.12037/) (2022)
- [AEGNN: Asynchronous Event-based Graph Neural Networks](https://mlsanity.com/paper/2203.17149/) (2022)
- [The rising costs of training frontier AI models](https://mlsanity.com/paper/2405.21015/) (2024)
- [LLM-FP4: 4-Bit Floating-Point Quantized Transformers](https://mlsanity.com/paper/2310.16836/) (2023)
- [Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of neoHebbian Three-Factor Learning Rules](https://mlsanity.com/paper/1801.05219/) (2018)
- [SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation](https://mlsanity.com/paper/2308.10873/) (2023)
- [Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies](https://mlsanity.com/paper/2304.12760/) (2023)
- [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://mlsanity.com/paper/2403.19888/) (2024)
- [Benchmarking Keyword Spotting Efficiency on Neuromorphic Hardware](https://mlsanity.com/paper/1812.01739/) (2018)
- [SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking](https://mlsanity.com/paper/2407.04752/) (2024)
- [A Spiking Neural Network for Image Segmentation](https://mlsanity.com/paper/2106.08921/) (2021)
- [An Introduction to Probabilistic Spiking Neural Networks: Probabilistic Models, Learning Rules, and Applications](https://mlsanity.com/paper/1910.01059/) (2019)
- [Low-Power Neuromorphic Hardware for Signal Processing Applications](https://mlsanity.com/paper/1901.03690/) (2019)
- [Neuroscience-inspired online unsupervised learning algorithms](https://mlsanity.com/paper/1908.01867/) (2019)

## Built on by (most cited first, 2 of 2)

- [Analog Quantum Asynchronous Event-Based Graph Neural Network](https://mlsanity.com/paper/2606.11000/) (2026)
- [Neuromorphic Diffusion Language Models: Addressing Compute and Memory Bottlenecks via Sparsity and Block Denoising](https://mlsanity.com/paper/2607.24841/) (2026)

## Abstract (arXiv)

The rapid growth of artificial intelligence (AI) has brought novel data processing and generative capabilities but also escalating energy requirements. This challenge motivates renewed interest in neuromorphic computing principles, which promise brain-like efficiency through discrete and sparse activations, recurrent dynamics, and non-linear feedback. In fact, modern AI architectures increasingly embody neuromorphic principles through heavily quantized activations, state-space dynamics, and sparse attention mechanisms. This paper elaborates on the connections between neuromorphic models, state-space models, and transformer architectures through the lens of the distinction between intra-token processing and inter-token processing. Most early work on neuromorphic AI was based on spiking neural networks (SNNs) for intra-token processing, i.e., for transformations involving multiple channels, or features, of the same vector input, such as the pixels of an image. In contrast, more recent research has explored how neuromorphic principles can be leveraged to design efficient inter-token processing methods, which selectively combine different information elements depending on their contextual relevance. Implementing associative memorization mechanisms, these approaches leverage state-space dynamics or sparse self-attention. Along with a systematic presentation of modern neuromorphic AI models through the lens of intra-token and inter-token processing, training methodologies for neuromorphic AI models are also reviewed. These range from surrogate gradients leveraging parallel convolutional processing to local learning rules based on reinforcement learning mechanisms.