On the Emergence and Test-Time Use of Structural Information in Large Language Models
Not on the map. Why?
Abstract (arXiv)
Learning structural information from observational data is central to producing new knowledge outside the training corpus. This holds for mechanistic understanding in scientific discovery as well as flexible test-time compositional generation. We thus study how language models learn abstract structures and utilize the learnt structural information at test-time. To ensure a controlled setup, we design a natural language dataset based on linguistic structural transformations. We empirically show that the emergence of learning structural information correlates with complex reasoning tasks, and that the ability to perform test-time compositional generation remains limited.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 25 Jan 2026.
Builds on
22 of 36 references.
References · 22
- 2017Attention Is All You Needcited by 59,932
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
- 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
- 2023How do Language Models Bind Entities in Context?cited by 102
- 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
- 2020Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networkscited by 72
- 2022Probing for the Usage of Grammatical Numbercited by 57
- 2022Causal Discovery in Heterogeneous Environments Under the Sparse Mechanism Shift Hypothesiscited by 54
- 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
10 more
- 2025Do-PFN: In-Context Learning for Causal Effect Estimationcited by 46
- 2022Causal de Finetti: On the Identification of Invariant Causal Structure in Exchangeable Datacited by 42
- 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
- 2022Variational Causal Dynamics: Discovering Modular World Models from Interventionscited by 16
- 2024Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learningcited by 16
- 2024Do Finetti: On Causal Effects for Exchangeable Datacited by 11
- 2023Out-of-Variable Generalization for Discriminative Modelscited by 4
- 2025Counterfactual reasoning: an analysis of in-context emergencecited by 4
- 2025Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learningcited by 4
- 2025Physics of Learning: A Lagrangian perspective to different learning paradigmscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.