Entropy-Generated Attention Beyond Softmax and Entmax: Kaniadakis and Reciprocal-Symmetric Abe Operators
Not on the map. Why?
Abstract (arXiv)
We derive two attention operators from generalized statistical entropies. Kaniadakis entropy yields an exact full-support normalization whose weights and low-score sensitivities decay algebraically, rather than exponentially as in Softmax or by exact truncation as in entmax. Classical Abe entropy yields an implicit reciprocal-symmetric operator. With q=eε, the involution qleftrightarrow q-1 removes every odd correction about Softmax; we obtain the normalized second- and fourth-order terms, including the deformation of the normalization multiplier. These stationary laws follow from a Fisher-metric Lagrangian on the probability simplex, whose Shannon sector recovers scaled dot-product Softmax. We also give a tangent-gradient test for deciding whether changing the entropy changes the attention profile or only its scale. R'enyi and two-parameter Sharma--Mittal entropies retain the Tsallis--entmax inverse-gradient shape, but their global moments make the effective temperature input dependent when the external temperature is fixed. Distinguishing profile-shape equivalence from fixed-parameter operator equivalence separates new normalization shapes from adaptive rescalings and organizes the operators by support, tail behavior, and realization complexity.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 11 of its 35 references that are in the map. First posted on arXiv 9 Feb 2026.
Builds on
11 of 35 references.
References · 11
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2018Neural Ordinary Differential Equationscited by 3,733
- 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
- 2019Sparse Sequence-to-Sequence Modelscited by 167
- 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
- 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
- 2017A Regularized Framework for Sparse and Structured Neural Attentioncited by 38
- 2018Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithmscited by 27
- 2025Scaled-Dot-Product Attention as One-Sided Entropic Optimal Transportcited by 7
- 2026Energy-Entropy Regularization: The True Power of Minimal Looped Transformerscited by 2
- 2026You Need Better Attention Priorscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.