# Beyond Attention: Toward Machines with Intrinsic Higher Mental States

arXiv: https://arxiv.org/abs/2505.06257 · HTML: https://mlsanity.com/paper/2505.06257/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 68 references that are in the map.
- First posted on arXiv 2 May 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Dendritic cortical microcircuits approximate the backpropagation algorithm](https://mlsanity.com/paper/1810.11393/) (2018)
- [Towards deep learning with segregated dendrites](https://mlsanity.com/paper/1610.00161/) (2016)
- [Local plasticity rules can learn deep representations using self-supervised contrastive predictions](https://mlsanity.com/paper/2010.08262/) (2020)
- [Single-phase deep learning in cortico-cortical networks](https://mlsanity.com/paper/2206.11769/) (2022)
- [The Dynamics of Functional Brain Networks: Integrated Network States during Cognitive Function](https://mlsanity.com/paper/1511.02976/) (2015)
- [A comparison of partial information decompositions using data from real and simulated layer 5b pyramidal cells](https://mlsanity.com/paper/2206.06456/) (2022)
- [Unlocking the potential of two-point cells for energy-efficient and resilient training of deep nets](https://mlsanity.com/paper/2211.01950/) (2022)
- [Cooperation Is All You Need](https://mlsanity.com/paper/2305.10449/) (2023)
- [Two-compartment neuronal spiking model expressing brain-state specific apical-amplification, -isolation and -drive regimes](https://mlsanity.com/paper/2311.06074/) (2023)
- [An Overlooked Role of Context-Sensitive Dendrites](https://mlsanity.com/paper/2408.11019/) (2024)

## Built on by (most cited first, 3 of 3)

- [Scalable Machines with Intrinsic Higher Mental-State Dynamics](https://mlsanity.com/paper/2603.13453/) (2026)
- [Single layer tiny Co4 outpaces GPT-2 and GPT-BERT](https://mlsanity.com/paper/2510.08404/) (2025)
- [A Cellular Doctrine of Morality: Intrinsic Active Precision and the Mind-Reality Overload Dilemma](https://mlsanity.com/paper/2605.01376/) (2026)

## Abstract (arXiv)

Attending to what is relevant is fundamental to both the mammalian brain and modern machine learning models such as Transformers. Yet, determining relevance remains a core challenge, traditionally offloaded to learning algorithms like backpropagation. Inspired by recent cellular neurobiological evidence linking neocortical pyramidal cells to distinct mental states, this work shows how models (e.g., Transformers) can emulate high-level perceptual processing and awake thought (imagination) states to pre-select relevant information before applying attention. Triadic neuronal-level modulation loops among questions (Q), clues (keys, K), and hypotheses (values, V) enable diverse, deep, parallel reasoning chains at the representation level and allow a rapid shift from initial biases to refined understanding. This leads to orders-of-magnitude faster learning with significantly reduced computational demand (e.g., fewer heads, layers, and tokens), at an approximate cost of O(N), where N is the number of input tokens. Results span reinforcement learning (e.g., CarRacing in a high-dimensional visual setup), computer vision, and natural language question answering.