# Transformers and Cortical Waves: Encoders for Pulling In Context Across Time

arXiv: https://arxiv.org/abs/2401.14267 · HTML: https://mlsanity.com/paper/2401.14267/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 131 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Hungry Hungry Hippos: Towards Language Modeling with State Space Models](https://mlsanity.com/paper/2212.14052/) (2022)
- [Large Language Models and the Reverse Turing Test](https://mlsanity.com/paper/2207.14382/) (2022)
- [Traveling Waves Encode the Recent Past and Enhance Sequence Learning](https://mlsanity.com/paper/2309.08045/) (2023)
- [Image segmentation with traveling waves in an exactly solvable recurrent neural network](https://mlsanity.com/paper/2311.16943/) (2023)
- [Internal feedback in the cortical perception-action loop enables fast and accurate behavior](https://mlsanity.com/paper/2211.05922/) (2022)
- [Language Modeling using LMUs: 10x Better Data Efficiency or Improved Scaling Compared to Transformers](https://mlsanity.com/paper/2110.02402/) (2021)

## Built on by (most cited first, 7 of 7)

- [Adaptive Intelligence: leveraging insights from adaptive behavior in animals to build flexible AI systems](https://mlsanity.com/paper/2411.15234/) (2024)
- [Improving the adaptive and continuous learning capabilities of artificial neural networks: Lessons from multi-neuromodulatory dynamics](https://mlsanity.com/paper/2501.06762/) (2025)
- [Fine-tuned network relies on generic representation to solve unseen cognitive task](https://mlsanity.com/paper/2406.18926/) (2024)
- [State Space Models Naturally Produce Time Cell and Oscillatory Behaviors and Scale to Abstract Cognitive Functions](https://mlsanity.com/paper/2507.13638/) (2025)
- [Dynamical Mechanisms for Coordinating Long-term Working Memory Based on the Precision of Spike-timing in Cortical Neurons](https://mlsanity.com/paper/2512.15891/) (2025)
- [Transformers self-organize like newborn visual systems when trained in prenatal worlds](https://mlsanity.com/paper/2601.03117/) (2026)
- [An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modeling](https://mlsanity.com/paper/2604.20595/) (2026)

## Abstract (arXiv)

The capabilities of transformer networks such as ChatGPT and other Large Language Models (LLMs) have captured the world's attention. The crucial computational mechanism underlying their performance relies on transforming a complete input sequence - for example, all the words in a sentence - into a long "encoding vector" that allows transformers to learn long-range temporal dependencies in naturalistic sequences. Specifically, "self-attention" applied to this encoding vector enhances temporal context in transformers by computing associations between pairs of words in the input sequence. We suggest that waves of neural activity traveling across single cortical areas or multiple regions at the whole-brain scale could implement a similar encoding principle. By encapsulating recent input history into a single spatial pattern at each moment in time, cortical waves may enable temporal context to be extracted from sequences of sensory inputs, the same computational principle used in transformers.