A completely uniform transformer for parity
Not on the map. Why?
Abstract (arXiv)
We construct a 3-layer constant-dimension transformer, recognizing the parity language, where neither parameter matrices nor the positional encoding depend on the input length. This improves upon a construction of Chiang and Cholak who use a positional encoding, depending on the input length (but their construction has 2 layers).
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 7 of its 8 references that are in the map. First posted on arXiv 5 Jan 2025.
Builds on
7 of 8 references.
References · 7
- 2017Attention Is All You Needcited by 60,361
- 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
- 2022Neural Networks and the Chomsky Hierarchycited by 217
- 2023Representational Strengths and Limitations of Transformerscited by 133
- 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
- 2024On Limitations of the Transformer Architecturecited by 63
- 2024Separations in the Representational Capabilities of Transformers and Recurrent Architecturescited by 29
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.