# Factorization Regret mediates compositional generalization in latent space

arXiv: https://arxiv.org/abs/2603.27134 · HTML: https://mlsanity.com/paper/2603.27134/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 124 references that are in the map.
- First posted on arXiv 28 Mar 2026.

## Builds on (most cited first, 23 of 23)

- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [In-context Reinforcement Learning with Algorithm Distillation](https://mlsanity.com/paper/2210.14215/) (2022)
- [ARC Prize 2024: Technical Report](https://mlsanity.com/paper/2412.04604/) (2024)
- [Conservative Objective Models for Effective Offline Model-Based Optimization](https://mlsanity.com/paper/2107.06882/) (2021)
- [Meta-learning of Sequential Strategies](https://mlsanity.com/paper/1905.03030/) (2019)
- [Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies](https://mlsanity.com/paper/1808.06508/) (2018)
- [Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents](https://mlsanity.com/paper/2102.02926/) (2021)
- [Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information](https://mlsanity.com/paper/1902.11239/) (2019)
- [Active Learning with Partial Feedback](https://mlsanity.com/paper/1802.07427/) (2018)
- [CompoSuite: A Compositional Reinforcement Learning Benchmark](https://mlsanity.com/paper/2207.04136/) (2022)
- [Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiences](https://mlsanity.com/paper/2509.16189/) (2025)
- [Dynamical Systems as Temporal Feature Spaces](https://mlsanity.com/paper/1907.06382/) (2019)
- [Aligned and oblique dynamics in recurrent neural networks](https://mlsanity.com/paper/2307.07654/) (2023)
- [Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks](https://mlsanity.com/paper/2210.02996/) (2022)
- [Interaction Information for Causal Inference: The Case of Directed Triangle](https://mlsanity.com/paper/1701.08868/) (2017)
- [Predictive Modeling in the Reservoir Kernel Motif Space](https://mlsanity.com/paper/2405.07045/) (2024)

## Abstract (arXiv)

Are there still barriers to generalization once all relevant variables are known? We consider the challenge of generalizing to a novel combination of task-relevant latent variables in the Cognitive Gridworld, a stationary Partially Observable Markov Decision Process where observations are generated by latent variables with parametric interactions. We establish Factorization Regret as an information-theoretic quantity measuring the contribution of latent variable interactions to task performance. Using this metric, we explain the sub-optimality of Echo State Networks and uncover a failure mode whereby confidence becomes decoupled from accuracy. This suggests that utilizing interactions between relevant variables is a non-trivial capability. We next introduce Representation Classification Chains to show that latent variable interactions and how to infer the variables' values can be learned together through Reinforcement Learning. Solving this variational inference problem enables generalization to novel combinations of relevant variables. Lastly, we demonstrate that once interactions are learned, imagined experience is sufficient for zero-shot control. In summary, we develop and motivate a minimal, theoretically grounded setting for research into generalization from experience.