Modes of Sequence Models and Learning Coefficients
Not on the map. Why?
Abstract (arXiv)
We develop a geometric account of sequence modelling that links patterns in the data to measurable properties of the loss landscape in transformer networks. First, we cast conditional sequence distributions into a Hilbert-space framework and apply tensor decompositions to identify their principal modes. Truncating the small-amplitude modes yields an effective data distribution that preserves dominant structure while discarding statistical detail. Second, we show theoretically that Local Learning Coefficient (LLC) estimates are insensitive to modes below a data-dependent threshold. Consequently, the LLC calculated in practice characterises the geometry of the effective rather than the true distribution. This insight clarifies why reliable LLC estimates can be obtained even when a network parameter is not a strict minimiser of the population loss, and it highlights how the inverse temperature in SGLD acts as a resolution dial on the landscape structure.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 26 of its 65 references that are in the map. First posted on arXiv 25 Apr 2025.
Builds on
26 of 65 references.
References · 26
- 2017Attention Is All You Needcited by 60,361
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
- 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
- 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
- 2015Character-Aware Neural Language Modelscited by 132
- 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
14 more
- 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 121
- 2024Transformers represent belief state geometry in their residual streamcited by 77
- 2022Formal Algorithms for Transformerscited by 72
- 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
- 2024Do language models plan ahead for future tokens?cited by 49
- 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
- 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 29
- 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 14
- 2024HLAT: High-quality Large Language Model Pre-trained on AWS Trainiumcited by 8
- 2013Optimal Kullback-Leibler Aggregation via Information Bottleneckcited by 4
- 2025Structure Development in List-Sorting Transformerscited by 4
- 2020On the Realization of Hidden Markov Models and Tensor Decompositioncited by 3
- 2022A Tutorial on the Spectral Theory of Markov Chainscited by 2
- 2024Low-Rank Tensors for Multi-Dimensional Markov Modelscited by 2
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.