mlsanityData to 14 Oct 2026

The Rank the Task Demands: A Causal Rank Law for Matrix Memories Trained on Group Composition

arXiv 2609.12259 ↗First posted 10 Sep 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Matrix-valued memories make rank the natural budget of a learned representation: the number of independent directions a state spans bounds what it can bind, compose, and track. We report causal evidence, on a group-composition testbed trained under a hard single-state bottleneck with a fixed decoder that cannot launder rank, that gradient descent recruits precisely the rank the task's algebra demands. A companion paper [Larson, 2026a] establishes the analogous recruitment and causal necessity pattern on a K-pair associative-binding testbed, where exact recovery provably requires state rank at least K; this paper inherits that instrument and extends the rank law from a scalar capacity bound to a representation-theoretic one. We train toward chosen minimal faithful reference representations embedded in larger matrices. On group-composition state tracking over five finite groups spanning the solvable/non-solvable divide, the recruited rank equals the group's minimal faithful real representation dimension dmin (Spearman ρ = 0.9747, the design's tie-capped maximum), the dimension-matched solvable/non-solvable pair S4/A5 is statistically equivalent under a pre-registered test, and a pre-registered force-rank test separates a guaranteed similarity ceiling from empirical recovery at the target dimension: one rank below dmin, cosine similarity is capped by the target's tied unit spectrum at √(dmin-1)/dmin ≤ 0.894, below the 0.9 threshold in every group by construction, with observed cells at 86-95% (mean 91%) of that ceiling; at dmin, not guaranteed a priori, recovery clears the pre-registered anchor-relative bar at four seeds per group in all five groups. Within this testbed, measured effective rank tracks representation dimension; the matched-dimension S4/A5 comparison establishes equivalence within the pre-registered tolerance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

23 of 29 references.

References · 23

  1. 2022Transformers Learn Shortcuts to Automatacited by 268
  2. 2022Neural Networks and the Chomsky Hierarchycited by 217
  3. 2024The Illusion of State in State-Space Modelscited by 187
  4. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
  5. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 101
  6. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 75
  7. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 51
  8. 2023Grokking Group Multiplication with Cosetscited by 24
  9. 2024Towards a unified and verified understanding of group-operation networkscited by 10
  10. 2026Sequential Group Composition: A Window into the Mechanics of Deep Learningcited by 10
  11. 2026The Expressive Limits of Diagonal SSMs for State-Trackingcited by 9
  12. 2026M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modelingcited by 8
11 more
  1. 2026On State Reduction in Linear Attentioncited by 6
  2. 2026When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformerscited by 6
  3. 2025Quantifying Memory Utilization with Effective State-Sizecited by 5
  4. 2025Learning words in groups: fusion algebras, tensor ranks and grokkingcited by 4
  5. 2026State Rank Dynamics in Linear Attention LLMscited by 3
  6. 2026Sharp Capacity Thresholds in Linear Associative Memory: From Top-1 Retrieval to Tail-Average Learningcited by 3
  7. 2026Neural Networks Provably Learn Spectral Representations for Group Compositioncited by 3
  8. 2026Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkincited by 2
  9. 2026An Algebraic View of the Expressivity of Recurrent Language Modelscited by 2
  10. 2026The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQAcited by 2
  11. 2026How to Tame Grokking: Representation Geometry as a Control Signalcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.