mlsanityData to 14 Oct 2026

Where Does Jagged Competence Come From?

arXiv 2610.03831 ↗First posted 2 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Capable systems often show jagged competence: low average error alongside failures on particular inputs. We ask where it comes from in a task built from two known layers. A lower layer A computes five per-slot sums from records; an upper layer B uses the slot-1 sum and a mode carried over from earlier boards to predict the next board's category, so B cannot be computed from the current A alone: B is a strict extension of A. We train small recurrent networks on B and measure, against exact ground truth, what they acquire of A. The central finding is that learning B gives the network a jagged version of A, measured through probe readability and supervised outputs. It is readable where B needs it (slot 1, 97-98% by a linear probe) and becomes less readable where B does not (the other slots, 5-10%); category misreads concentrate near the slot-1 cutoffs; and when A is trained explicitly it comes out only approximately right. The networks show jagged competence in B: natural KL below 3×10-4 bits coexists with a maximum law TV of about 0.25 on constructed histories. Matched experiments show that even a good A is not enough: connecting a learned A to B cuts misreads 1.3 to 17-fold, the same exact A gives fewer misreads as one-hot inputs (0-2) than as numerical inputs (10-317), one B update makes an exact A inexact unless the B gradient is blocked, and exact A still leaves some B failures. In this task, uneven acquisition, access and preservation of the lower theory explain part of the jagged competence of a network that learns the theory built on it; some failures remain unexplained. Whether the same holds in larger systems is a test to run.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

26 of 37 references.

References · 26

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  5. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  6. 2022Toy Models of Superpositioncited by 971
  7. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  8. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  9. 2019Designing and Interpreting Probes with Control Taskscited by 640
  10. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  11. 2021Causal Abstractions of Neural Networkscited by 422
  12. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
14 more
  1. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 267
  2. 2022Neural Networks and the Chomsky Hierarchycited by 217
  3. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  4. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
  5. 2020What shapes feature representations? Exploring datasets, architectures, and trainingcited by 126
  6. 2023Calibrated Language Models Must Hallucinatecited by 115
  7. 2024Evaluating the World Model Implicit in a Generative Modelcited by 113
  8. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 108
  9. 2026Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representationscited by 17
  10. 2026A Model of Artificial Jagged Intelligencecited by 2
  11. 2026Six Birds: Foundations of Emergence Calculuscited by 1
  12. 2026Arithmetic OOD Failure Unfolds in Stages in Minimal GPTscited by 1
  13. 2026Artificial Jagged Intelligence as Uneven Optimization Energy Allocation Capability Concentration, Redistribution, and Optimization Governancecited by 1
  14. 2026Represented Is Not Computed: A Causal Test of Candidate Algorithmic Intermediates in a Transformercited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Operationalising the Superficial Alignment Hypothesis via Task Complexity
  2. 2026Pre-carved Niches: The Formation Dynamics of Modular Task Partitions in Early LLM Training
  3. 2025How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients
  4. 2026Linear Separability of Activation Representations after Supervised Fine-Tuning on Incorrect Responses: A Study of Synthetic Dishonesty in Large Language Models
  5. 2024Revisiting the Superficial Alignment Hypothesis
  6. 2026The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context
  7. 2025Characterising the Inductive Biases of Neural Networks on Boolean Data
  8. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesis