mlsanityData to 7 Oct 2026

Entropy-Generated Attention Beyond Softmax and Entmax: Kaniadakis and Reciprocal-Symmetric Abe Operators

arXiv 2602.08216 ↗First posted 9 Feb 2026cs.LG cond-mat.stat-mech stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

We derive two attention operators from generalized statistical entropies. Kaniadakis entropy yields an exact full-support normalization whose weights and low-score sensitivities decay algebraically, rather than exponentially as in Softmax or by exact truncation as in entmax. Classical Abe entropy yields an implicit reciprocal-symmetric operator. With q=eε, the involution qleftrightarrow q-1 removes every odd correction about Softmax; we obtain the normalized second- and fourth-order terms, including the deformation of the normalization multiplier. These stationary laws follow from a Fisher-metric Lagrangian on the probability simplex, whose Shannon sector recovers scaled dot-product Softmax. We also give a tangent-gradient test for deciding whether changing the entropy changes the attention profile or only its scale. R'enyi and two-parameter Sharma--Mittal entropies retain the Tsallis--entmax inverse-gradient shape, but their global moments make the effective temperature input dependent when the external temperature is fixed. Distinguishing profile-shape equivalence from fixed-parameter operator equivalence separates new normalization shapes from adaptive rescalings and organizes the operators by support, tail behavior, and realization complexity.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 11 of its 35 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

11 of 35 references.

References · 11

  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  4. 2019Sparse Sequence-to-Sequence Modelscited by 167
  5. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  6. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  7. 2017A Regularized Framework for Sparse and Structured Neural Attentioncited by 38
  8. 2018Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithmscited by 27
  9. 2025Scaled-Dot-Product Attention as One-Sided Entropic Optimal Transportcited by 7
  10. 2026Energy-Entropy Regularization: The True Power of Minimal Looped Transformerscited by 2
  11. 2026You Need Better Attention Priorscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.