# From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers

arXiv: https://arxiv.org/abs/2602.06923 · HTML: https://mlsanity.com/paper/2602.06923/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Discovering Symbolic Models from Deep Learning with Inductive Biases](https://mlsanity.com/paper/2006.11287/) (2020)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [AI Poincar'e: Machine Learning Conservation Laws from Trajectories](https://mlsanity.com/paper/2011.04698/) (2020)
- [Rediscovering orbital mechanics with machine learning](https://mlsanity.com/paper/2202.02306/) (2022)
- [Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2502.20388/) (2025)
- [What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models](https://mlsanity.com/paper/2507.06952/) (2025)
- [AI Poincar'e 2.0: Machine Learning Conservation Laws from Differential Equations](https://mlsanity.com/paper/2203.12610/) (2022)
- [On the Emergence of Linear Analogies in Word Embeddings](https://mlsanity.com/paper/2505.18651/) (2025)

## Built on by (most cited first, 7 of 7)

- [stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation](https://mlsanity.com/paper/2605.21800/) (2026)
- [The Observer Effect in World Models: Invasive Adaptation Corrupts Latent Physics](https://mlsanity.com/paper/2602.12218/) (2026)
- [Beyond Myopic World Models: Long-Horizon End-to-End Training for Direct Future Prediction](https://mlsanity.com/paper/2608.07420/) (2026)
- [Revisiting Euler-Angle Regression with Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2607.09650/) (2026)
- [Extracting Algorithms in Pre-trained LLMs: A Case on Hidden Markov Models](https://mlsanity.com/paper/2607.22646/) (2026)
- [LAWFUL: Law-Aligned Witness for Faithful Use of Latents](https://mlsanity.com/paper/2607.28672/) (2026)
- [Why Do Conventional World Models Fail to Learn Cellular Automata?](https://mlsanity.com/paper/2609.39604/) (2026)

## Abstract (arXiv)

Can general-purpose AI architectures go beyond prediction to discover the physical laws governing the universe? True intelligence relies on "world models" -- causal abstractions that allow an agent to not only predict future states but understand the underlying governing dynamics. While previous "AI Physicist" approaches have successfully recovered such laws, they typically rely on strong, domain-specific priors that effectively "bake in" the physics. Conversely, Vafa et al. recently showed that generic Transformers fail to acquire these world models, achieving high predictive accuracy without capturing the underlying physical laws. We bridge this gap by systematically introducing three minimal inductive biases. We show that ensuring spatial smoothness (by formulating prediction as continuous regression) and stability (by training with noisy contexts to mitigate error accumulation) enables generic Transformers to surpass prior failures and learn a coherent Keplerian world model, successfully fitting ellipses to planetary trajectories. However, true physical insight requires a third bias: temporal locality. By restricting the attention window to the immediate past -- imposing the simple assumption that future states depend only on the local state rather than a complex history -- we force the model to abandon curve-fitting and discover Newtonian force representations. Our results demonstrate that simple architectural choices determine whether an AI becomes a curve-fitter or a physicist, marking a critical step toward automated scientific discovery.