# From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics

arXiv: https://arxiv.org/abs/2608.21174 · HTML: https://mlsanity.com/paper/2608.21174/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 21 Aug 2026.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [Query-Key Normalization for Transformers](https://mlsanity.com/paper/2010.04245/) (2020)
- [When Attention Sink Emerges in Language Models: An Empirical View](https://mlsanity.com/paper/2410.10781/) (2024)
- [Sparse Sequence-to-Sequence Models](https://mlsanity.com/paper/1905.05702/) (2019)
- [Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing](https://mlsanity.com/paper/2306.12929/) (2023)
- [CoLT5: Faster Long-Range Transformers with Conditional Computation](https://mlsanity.com/paper/2303.09752/) (2023)
- [Theory, Analysis, and Best Practices for Sigmoid Self-Attention](https://mlsanity.com/paper/2409.04431/) (2024)
- [Sparse Attention with Linear Units](https://mlsanity.com/paper/2104.07012/) (2021)
- [Outlier-Efficient Hopfield Layers for Large Transformer-Based Models](https://mlsanity.com/paper/2404.03828/) (2024)
- [Softpick: No Attention Sink, No Massive Activations with Rectified Softmax](https://mlsanity.com/paper/2504.20966/) (2025)
- [Mask-Align: Self-Supervised Neural Word Alignment](https://mlsanity.com/paper/2012.07162/) (2020)
- [StableMask: Refining Causal Masking in Decoder-only Transformer](https://mlsanity.com/paper/2402.04779/) (2024)
- [Learning to Skip for Language Modeling](https://mlsanity.com/paper/2311.15436/) (2023)
- [Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks](https://mlsanity.com/paper/2603.11487/) (2026)
- [Not Another Imputation Method: A Transformer-based Model for Missing Values in Tabular Datasets](https://mlsanity.com/paper/2407.11540/) (2024)
- [Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers](https://mlsanity.com/paper/2510.09017/) (2025)
- [Zero Token-Driven Deep Thinking in LLMs: Unlocking the Full Potential of Existing Parameters via Cyclic Refinement](https://mlsanity.com/paper/2502.12214/) (2025)
- [QUEST: A robust attention formulation using query-modulated spherical attention](https://mlsanity.com/paper/2604.00199/) (2026)

## Abstract (arXiv)

Attention masks are relation-level controls: they specify which query--source pairs may interact. They do not provide a representation-carried token state that is non-participating at the attention boundary. We assign each token hidden carrier (hi) an active-presence coefficient (pi=lVert hirVert2/(τ+lVert hirVert2)). The same coefficient has two roles: it gates information emitted by token (i), and it determines the mass with which token (i) enters computations shared with other tokens. OAttention is the support-coupled attention realization of this rule. It gates the receiver output by (pi) and weights source (j) by (pj) in both the attention numerator and partition, while retaining the standard score, visibility relation, exponential competition, and value aggregation. This makes the zero-vector token a zero element and yields exact null-receiver, null-source insertion, self-attention insertion, and empty-support properties. The same token-level presence gives local O-components (OFFN, ONorm, and OInject), presence-weighted OStandardize, the O-Closure law (M(Hoplus0)=M(H)oplus0), and an OTransformer by residual and compositional closure. The canonical operator is checked by contract tests and a GPU evaluation. In a zero-fine-tuning retrofit of a cloned pretrained TabPFN v3 regressor, calibrated hidden-carrier OAttention and Full-O variants change mean RMSE by +0.088% and +0.177%, respectively, over 18 matched dataset--seed cases. A two-block ablation shows that OAttention alone does not preserve a NULL state through ordinary host components, whereas the OTransformer path does. These are scoped tests of exactness, active-path compatibility, and compositional necessity; they do not establish universal no-loss, arbitrary-host closure, learned attraction to the origin, or a general semantics for missing values.