mlsanityData to 13 Oct 2026

Sequences of Logits Reveal the Low Rank Structure of Language Models

arXiv 2510.24966 ↗First posted 28 Oct 2025cs.LG cs.AI cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

A major problem in the study of large language models is to understand their inherent low-dimensional structure. We introduce an approach to study the low-dimensional structure of language models at a model-agnostic level: as sequential probabilistic models. We first empirically demonstrate that a wide range of modern language models exhibit low-rank structure: in particular, matrices built from the model's logits for varying sets of prompts and responses have low approximate rank. We then show that this low-rank structure can be leveraged for generation -- in particular, we can generate a response to a target prompt using a linear combination of the model's outputs on unrelated, or even nonsensical prompts. On the theoretical front, we observe that studying the approximate rank of language models in the sense discussed above yields a simple universal abstraction whose theoretical predictions parallel our experiments. We then analyze the representation power of the abstraction and give provable learning guarantees.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Builds on 43 of its 70 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

43 of 70 references.

References · 43

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,658
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,064
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,650
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,440
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,994
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,637
  8. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,307
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,009
  10. 2022In-context Learning and Induction Headscited by 940
  11. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 706
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 665
31 more
  1. 2015Generating Sentences from a Continuous Spacecited by 609
  2. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  3. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 529
  4. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 421
  5. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  6. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 347
  7. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 316
  8. 2023In-Context Learning Creates Task Vectorscited by 305
  9. 2023Function Vectors in Large Language Modelscited by 299
  10. 2016Using the Output Embedding to Improve Language Modelscited by 269
  11. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 229
  12. 2013Predicting Parameters in Deep Learningcited by 224
  13. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  14. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 195
  15. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 191
  16. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 165
  17. 2024Stealing Part of a Production Language Modelcited by 157
  18. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 148
  19. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  20. 2014word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding methodcited by 112
  21. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  22. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 50
  23. 2024Logits of API-Protected LLMs Leak Proprietary Informationcited by 38
  24. 2023Closing the Curious Case of Neural Text Degenerationcited by 30
  25. 2024One-layer transformers fail to solve the induction heads taskcited by 30
  26. 2023Spectral State Space Modelscited by 20
  27. 2018Sigsoftmax: Reanalysis of the Softmax Bottleneckcited by 19
  28. 2024Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneckcited by 19
  29. 2019Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearitiescited by 12
  30. 2023Learning Hidden Markov Models Using Conditional Samplescited by 12
  31. 2024Model Stealing for Any Low-Rank Language Modelcited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026Subliminal Effects in Your Data: A General Mechanism via Log-Linearitycited by 17
  2. 2025Provably Learning from Modern Language Models via Low Logit Rankcited by 3
  3. 2026Reject, Resample, Repeat: Understanding Parallel Reasoning in Language Model Inferencecited by 3
  4. 2025Provably Extracting the Features from a General Superpositioncited by 1
  5. 2026Identified-Set Geometry of Distributional Model Extraction under Top-K Censored API Accesscited by 1
  6. 2026Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillationcited by 1
  7. 2026LoRi: Low-Rank Distillation for Implicit Reasoningcited by 0
  8. 2026How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attentioncited by 0