# Deriving Transformer Architectures as Implicit Multinomial Regression

arXiv: https://arxiv.org/abs/2509.04653 · HTML: https://mlsanity.com/paper/2509.04653/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 1 of its 8 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Builds on (most cited first, 1 of 1)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)

## Abstract (arXiv)

While attention has been empirically shown to improve model performance, it lacks a rigorous mathematical justification. This short paper establishes a novel connection between attention mechanisms and multinomial regression. Specifically, we show that in a fixed multinomial regression setting, optimizing over latent features yields solutions that align with the dynamics induced on features by attention blocks. In other words, the evolution of representations through a transformer can be interpreted as a trajectory that recovers the optimal features for classification.