# Named Tensor Notation

arXiv: https://arxiv.org/abs/2102.13196 · HTML: https://mlsanity.com/paper/2102.13196/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 34 references that are in the map.
- First posted on arXiv 25 Feb 2021.

## Builds on (most cited first, 3 of 3)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)

## Built on by (most cited first, 11 of 11)

- [Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.15647/) (2023)
- [Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architectures](https://mlsanity.com/paper/2402.05424/) (2024)
- [Penzai + Treescope: A Toolkit for Interpreting, Visualizing, and Editing Models As Data](https://mlsanity.com/paper/2408.00211/) (2024)
- [Revisiting File Context for Source Code Summarization](https://mlsanity.com/paper/2309.02326/) (2023)
- [Towards a Categorical Foundation of Deep Learning: A Survey](https://mlsanity.com/paper/2410.05353/) (2024)
- [Broadcast Product: Redefining Shape-aligned Element-wise Multiplication and Beyond](https://mlsanity.com/paper/2409.17502/) (2024)
- [Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks](https://mlsanity.com/paper/2501.05930/) (2025)
- [Operon: Incremental Construction of Ragged Data via Named Dimensions](https://mlsanity.com/paper/2511.16080/) (2025)
- [Weaves, Wires, and Morphisms: Formalizing and Implementing the Algebra of Deep Learning](https://mlsanity.com/paper/2604.07242/) (2026)
- [It's All Just Vectorization: einx, a Universal Notation for Tensor Operations](https://mlsanity.com/paper/2607.27987/) (2026)
- [Graphical Design of Interpretable Architectures](https://mlsanity.com/paper/2608.18936/) (2026)

## Abstract (arXiv)

We propose a notation for tensors with named axes, which relieves the author, reader, and future implementers of machine learning models from the burden of keeping track of the order of axes and the purpose of each. The notation makes it easy to lift operations on low-order tensors to higher order ones, for example, from images to minibatches of images, or from an attention mechanism to multiple attention heads. After a brief overview and formal definition of the notation, we illustrate it through several examples from modern machine learning, from building blocks like attention and convolution to full models like Transformers and LeNet. We then discuss differential calculus in our notation and compare with some alternative notations. Our proposals build on ideas from many previous papers and software libraries. We hope that our notation will encourage more authors to use named tensors, resulting in clearer papers and more precise implementations.