mlsanityData to 14 Oct 2026

What a Cross-Model Fixed-Point Census Can and Cannot Arbitrate About Repetition

arXiv 2609.29507 ↗First posted 24 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Two accounts of neural text degeneration coexist. One locates the cause in the training data -- repetition in the corpus produces repetition in the output, established by training on repetition-sorted data -- the other in the trained network, in copying circuits and repetition features. Neither has been arbitrated across a broad cohort of pretrained models: the causal work trains its own. We report an observational measurement in a different currency: the fixed-point structure of a model's own short-window argmax map, censused from 96 random two-token starts over 17 off-the-shelf models, always unprompted -- a companion paper shows nine tokens of conditioning move this readout across most of its range. The four-way class is stable across census seeds on 17 of 17. Three exhibits. At fixed corpus (The Pile), fixed scale and that fixed domain, the class is not determined: across two size-matched tiers, pythia is a funnel while RWKV, Mamba and a second transformer family are not, and both hold their class across an order of magnitude of scale. Six of seven models in that ladder reach the same endpoint token, and those concentrating on it most strongly are among those that never stay there -- what varies is not where trajectories go but whether the destination self-continues. The deduplicated Pythia suite does not change the class. And the corpus-side inflow term proposed for this phenomenon does not select our endpoints once frequency is controlled, in English and three other languages. This is observational and cannot refute a training intervention. Funnels are common: eight of seventeen models, seven families, five corpora -- so the limit is not that the phenomenon is one model's peculiarity, but that within the one corpus where training data can be held fixed only one available family funnels; that subset cannot show the split is corpus-independent.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 10 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

10 of 10 references.

References · 10

  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  3. 2023Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspectivecited by 39
  4. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architecturescited by 31
  5. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  6. 2025Predicting the Emergence of Induction Heads in Language Model Pretrainingcited by 7
  7. 2024Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metricscited by 6
  8. 2026What Iterated Self-Feeding Probes of Language Models Measure, and a test that separates the construction from the modelcited by 3
  9. 2025Correlation Dimension of Auto-Regressive Large Language Modelscited by 2
  10. 2026Prompt-Model Interaction Reaches the Fixed Points: A deterministic, task-free structural readout -- and the factorizations of it that failedcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Break Step: Recursive Training Resonates with Replayed Sampling Noise
  2. 2026Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs
  3. 2022Scaling Laws and Interpretability of Learning from Repeated Data
  4. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
  5. 2026History Replays Itself: Unleashing Long-Context Capability in Linear Attention
  6. 2026To Copy or Not to Copy: Copying Is Easier to Induce Than Recall
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
  8. 2026Linguistic properties and model scale in brain encoding: from small to compressed language models