mlsanityData to 7 Oct 2026

Lines of Thought in Large Language Models

arXiv 2410.01545 ↗First posted 2 Oct 2024cs.LG physics.data-ancited by 5

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large Language Models achieve next-token prediction by transporting a vectorized piece of text (prompt) across an accompanying embedding space under the action of successive transformer layers. The resulting high-dimensional trajectories realize different contextualization, or 'thinking', steps, and fully determine the output probability distribution. We aim to characterize the statistical properties of ensembles of these 'lines of thought.' We observe that independent trajectories cluster along a low-dimensional, non-Euclidean manifold, and that their path can be well approximated by a stochastic equation with few parameters extracted from data. We find it remarkable that the vast complexity of such large models can be reduced to a much simpler form, and we reflect on implications.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 31 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

17 of 31 references.

References · 17

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2023Mistral 7Bcited by 3,065
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  7. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  9. 2023Language Models Represent Space and Timecited by 306
  10. 2023The geometry of hidden representations of large transformer modelscited by 108
  11. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  12. 2024Amortized Planning with Large-Scale Transformers: A Case Study on Chesscited by 45
5 more
  1. 2024LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling lawcited by 32
  2. 2023Uncovering hidden geometry in Transformers via disentangling position and contextcited by 15
  3. 2024Transformer Block Coupling and its Correlation with Generalization in LLMscited by 9
  4. 2019Visualizing Attention in Transformer-Based Language Representation Modelscited by 8
  5. 2024Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  2. 2026The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriorscited by 15
  3. 2025Entropy-Lens: Uncovering Decision Strategies in LLMscited by 6
  4. 2026The Depth Flow of Token Representations Is Nonlinear and Does Not Descend Its Own Densitycited by 0
  5. 2026Measuring Optimal Transport in Transformer Depthcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Probabilistic Structure of Large Language Models
  2. 2025Modeling Language as a Sequence of Thoughts
  3. 2025Correlation Dimension of Auto-Regressive Large Language Models
  4. 2023A Latent Space Theory for Emergent Abilities in Large Language Models
  5. 2023Why Can Large Language Models Generate Correct Chain-of-Thoughts?
  6. 2026Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoning
  7. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction
  8. 2025A Markov Categorical Framework for Language Modeling