# Entropy-Generated Attention Beyond Softmax and Entmax: Kaniadakis and Reciprocal-Symmetric Abe Operators

arXiv: https://arxiv.org/abs/2602.08216 · HTML: https://mlsanity.com/paper/2602.08216/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 35 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Builds on (most cited first, 11 of 11)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Sparse Sequence-to-Sequence Models](https://mlsanity.com/paper/1905.05702/) (2019)
- [Stabilizing Transformer Training by Preventing Attention Entropy Collapse](https://mlsanity.com/paper/2303.06296/) (2023)
- [Sinkformers: Transformers with Doubly Stochastic Attention](https://mlsanity.com/paper/2110.11773/) (2021)
- [A Regularized Framework for Sparse and Structured Neural Attention](https://mlsanity.com/paper/1705.07704/) (2017)
- [Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms](https://mlsanity.com/paper/1805.09717/) (2018)
- [Scaled-Dot-Product Attention as One-Sided Entropic Optimal Transport](https://mlsanity.com/paper/2508.08369/) (2025)
- [You Need Better Attention Priors](https://mlsanity.com/paper/2601.15380/) (2026)
- [Energy-Entropy Regularization: The True Power of Minimal Looped Transformers](https://mlsanity.com/paper/2601.09588/) (2026)

## Built on by (most cited first, 2 of 2)

- [Thermodynamic Weight Decay: Exploring Grokking Acceleration via Attention Specific Heat](https://mlsanity.com/paper/2607.20552/) (2026)
- [Spectral Origins of the Self-Correction Blind Spot in Autoregressive Generation](https://mlsanity.com/paper/2607.09803/) (2026)

## Abstract (arXiv)

We derive two attention operators from generalized statistical entropies. Kaniadakis entropy yields an exact full-support normalization whose weights and low-score sensitivities decay algebraically, rather than exponentially as in Softmax or by exact truncation as in entmax. Classical Abe entropy yields an implicit reciprocal-symmetric operator. With q=eε, the involution qleftrightarrow q-1 removes every odd correction about Softmax; we obtain the normalized second- and fourth-order terms, including the deformation of the normalization multiplier. These stationary laws follow from a Fisher-metric Lagrangian on the probability simplex, whose Shannon sector recovers scaled dot-product Softmax. We also give a tangent-gradient test for deciding whether changing the entropy changes the attention profile or only its scale. R'enyi and two-parameter Sharma--Mittal entropies retain the Tsallis--entmax inverse-gradient shape, but their global moments make the effective temperature input dependent when the external temperature is fixed. Distinguishing profile-shape equivalence from fixed-parameter operator equivalence separates new normalization shapes from adaptive rescalings and organizes the operators by support, tail behavior, and realization complexity.