mlsanityData to 14 Oct 2026

A Taxonomy of Transcendence

arXiv 2508.17669 ↗First posted 25 Aug 2025cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Although language models are trained to mimic humans, the resulting systems display capabilities beyond the scope of any one person. To understand this phenomenon, we use a controlled setting to identify properties of the training data that lead a model to transcend the performance of its data sources. We build on previous work to outline three modes of transcendence, which we call skill denoising, skill selection, and skill generalization. We then introduce a knowledge graph-based setting in which simulated experts generate data based on their individual expertise. We highlight several aspects of data diversity that help to enable the model's transcendent capabilities. Additionally, our data generation setting offers a controlled testbed that we hope is valuable for future research in the area.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 25 Aug 2025.

Builds on

24 of 34 references.

References · 24

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  6. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  7. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  8. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
  9. 2024More Agents Is All You Needcited by 180
  10. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  11. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
  12. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
12 more
  1. 2024Knowledge Fusion of Large Language Modelscited by 115
  2. 2023Why think step by step? Reasoning emerges from the locality of experiencecited by 103
  3. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  4. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  5. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 60
  6. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  7. 2021Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalizationcited by 27
  8. 2023Diversity of Thought Improves Reasoning Abilities of LLMscited by 25
  9. 2024Transcendence: Generative Models Can Outperform The Experts That Train Themcited by 20
  10. 2024ALPINE: Unveiling the Planning Capability of Autoregressive Learning in Language Modelscited by 18
  11. 2023Attribute Diversity Determines the Systematicity Gap in VQAcited by 3
  12. 2023D3: Data Diversity Design for Systematic Generalization in Visual Question Answeringcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiencescited by 15
  2. 2025Out-of-distribution Tests Reveal Compositionality in Chess Transformerscited by 2