mlsanityData to 7 Oct 2026

Input Pathways Shape Few-Shot, Not Zero-Shot, Binding in Tiny Transformers: A Fully-Enumerable Study

arXiv 2607.04926 ↗First posted 6 Jul 2026cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

How does the way information reaches a transformer -- as symbolic tokens, a clean per-factor "oracle" code, or an entangled perceptual vector -- shape whether it binds that information compositionally? We study ~6-10K-parameter transformers on finite factored worlds enumerated exhaustively, so every measurement covers the whole input space (zero sampling variance) and the informative routes are information-matched (exact Bayes ceiling 1.0). We report four findings. (1) Endpoint invariance: on held-out binding queries no informative route reaches converged zero-shot composition -- each ends at or below chance despite a ceiling of 1.0, so within a bounded sweep the failure reflects inductive bias under a lookup-sufficient objective, not missing information. (2) A two-factor account of few-shot binding: sample efficiency is best explained by input-pathway parameter sharing and code readability; a dimension-matched control and a graded readability sweep isolate readability from input dimension, and the clean oracle is not the most sample-efficient readable route. (3) A double dissociation: early in training, distributed -- but not index-like -- codes pass through a transient above-chance phase (tracking code format), while few-shot efficiency tracks pathway sharing. (4) Failure anatomy: symbolic routes lose the answer at the readout; index routes mis-bind (the answer stays decodable, yet an input intervention shows the output tracks the wrong slot); entangled routes inherit their input's readability. The central claim is the two-factor account; the endpoint and anatomy results are diagnostic constraints. All code, manifests, and per-seed logs are released for exact reproduction.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 29 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

24 of 29 references.

References · 24

  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  2. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  3. 2022Toy Models of Superpositioncited by 928
  4. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  5. 2019Designing and Interpreting Probes with Control Taskscited by 608
  6. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  7. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  8. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  10. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  11. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  12. 2020Experience Grounds Languagecited by 241
12 more
  1. 2020On the Binding Problem in Artificial Neural Networkscited by 230
  2. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  3. 2020What shapes feature representations? Exploring datasets, architectures, and trainingcited by 125
  4. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  5. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
  6. 2023How do Language Models Bind Entities in Context?cited by 102
  7. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  8. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  9. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  10. 2024Case-Based or Rule-Based: How Do Transformers Do the Math?cited by 26
  11. 2024Compact Proofs of Model Performance via Mechanistic Interpretabilitycited by 17
  12. 2025The Mechanistic Emergence of Symbol Grounding in Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026What You Can't See Is What You Learn: Slot-Selective Evidence Masking Favors Compositional Generalization in Shared-Genome Language-Model Societiescited by 1
  2. 2026Small Enough to Know Everything: The Fully-Enumerable Transformer as an Instrument for the Science of Delayed Generalizationcited by 0