The Predictive-Causal Gap: An Impossibility Theorem and Large-Scale Neural Evidence
Not on the map. Why?
Abstract (arXiv)
We report a systematic failure mode in predictive representation learning. Across 2695 neural network configurations trained to predict linear-Gaussian dynamics, the optimal encoder tracks the environment rather than the system it is meant to model. The mean causal fidelity -- the fraction of encoder sensitivity allocated to system degrees of freedom -- is 0.49, and only 2.5% of configurations exceed 0.70. The failure intensifies with dimension: at N=100, the optimal encoder becomes causally blind (fidelity ~10^{-8}) while achieving 92% lower prediction error than the causal representation. We prove this is not an optimization artifact but a structural property of the predictive objective: when environment modes are slower or less noisy than system modes, every minimizer of the population risk encodes the former. The set of dynamics exhibiting this predictive-causal gap is open and of positive measure in parameter space. In a nonlinear Duffing-GRU sweep, unconstrained predictors learn environment-dominant representations in 55% of tasks (95% CI 41--68%) versus 24% under operational grounding (p=2.3e-3); the median out-of-distribution MSE inflation under environment shift is 1.82x versus 1.00x. Operational grounding -- restricting the loss to system observables -- partially suppresses the gap, but causal fidelity is never recovered without an explicit system-environment boundary. The results identify the predictive-causal gap as a structural limit of learning, with implications for self-supervised representation learning, world models, and the scaling paradigm.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 4 of its 8 references that are in the map. First posted on arXiv 6 May 2026.
Builds on
4 of 8 references.
References · 4
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
- 2023Mastering Diverse Domains through World Modelscited by 1,253
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Prediction horizon shapes representations in predictive learning
- 2026I Act Therefore I Am: When Is JEPA's Action-Conditioning Enough to Learn Causal Mechanisms?
- 2026Representational Simplicity and Circuit Size Dissociate in a Threshold-Dependent Way: A Controlled Test via Adversarial Training
- 2026Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective
- 2026From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness
- 2026What to Preserve in Recursive Computation: A Local Predictive Sufficiency Principle
- 2026Predictive Memory Localization: Forecasting Selective Intervention Paths from Internal Signals
- 2026AVL-JEPA: Preventing Causal Dynamics Information Collapse In Joint Embedding Predictive Architecture World Models