mlsanityData to 7 Oct 2026

From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics

arXiv 2608.21174 ↗First posted 21 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Attention masks are relation-level controls: they specify which query--source pairs may interact. They do not provide a representation-carried token state that is non-participating at the attention boundary. We assign each token hidden carrier (hi) an active-presence coefficient (pi=lVert hirVert2/(τ+lVert hirVert2)). The same coefficient has two roles: it gates information emitted by token (i), and it determines the mass with which token (i) enters computations shared with other tokens. OAttention is the support-coupled attention realization of this rule. It gates the receiver output by (pi) and weights source (j) by (pj) in both the attention numerator and partition, while retaining the standard score, visibility relation, exponential competition, and value aggregation. This makes the zero-vector token a zero element and yields exact null-receiver, null-source insertion, self-attention insertion, and empty-support properties. The same token-level presence gives local O-components (OFFN, ONorm, and OInject), presence-weighted OStandardize, the O-Closure law (M(Hoplus0)=M(H)oplus0), and an OTransformer by residual and compositional closure. The canonical operator is checked by contract tests and a GPU evaluation. In a zero-fine-tuning retrofit of a cloned pretrained TabPFN v3 regressor, calibrated hidden-carrier OAttention and Full-O variants change mean RMSE by +0.088% and +0.177%, respectively, over 18 matched dataset--seed cases. A two-block ablation shows that OAttention alone does not preserve a NULL state through ordinary host components, whereas the OTransformer path does. These are scoped tests of exactness, active-path compatibility, and compositional necessity; they do not establish universal no-loss, arbitrary-host closure, learned attraction to the origin, or a general semantics for missing values.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

24 of 32 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2017Deep Setscited by 1,394
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  5. 2017Dynamic Routing Between Capsulescited by 689
  6. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  7. 2020Query-Key Normalization for Transformerscited by 266
  8. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  9. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  10. 2019Sparse Sequence-to-Sequence Modelscited by 167
  11. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  12. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
12 more
  1. 2024Theory, Analysis, and Best Practices for Sigmoid Self-Attentioncited by 62
  2. 2021Sparse Attention with Linear Unitscited by 41
  3. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  4. 2020Mask-Align: Self-Supervised Neural Word Alignmentcited by 20
  5. 2024StableMask: Refining Causal Masking in Decoder-only Transformercited by 20
  6. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 20
  7. 2023Learning to Skip for Language Modelingcited by 19
  8. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 10
  9. 2024Not Another Imputation Method: A Transformer-based Model for Missing Values in Tabular Datasetscited by 9
  10. 2025Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformerscited by 7
  11. 2025Zero Token-Driven Deep Thinking in LLMs: Unlocking the Full Potential of Existing Parameters via Cyclic Refinementcited by 5
  12. 2026QUEST: A robust attention formulation using query-modulated spherical attentioncited by 1

Built on by

Not cited yet.