# What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models

arXiv: https://arxiv.org/abs/2507.06952 · HTML: https://mlsanity.com/paper/2507.06952/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 39 of its 69 references that are in the map.
- First posted on arXiv 9 Jul 2025.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Discovering Symbolic Models from Deep Learning with Inductive Biases](https://mlsanity.com/paper/2006.11287/) (2020)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Transformers Learn Shortcuts to Automata](https://mlsanity.com/paper/2210.10749/) (2022)
- [The Parallelism Tradeoff: Limitations of Log-Precision Transformers](https://mlsanity.com/paper/2207.00729/) (2022)
- [The Illusion of State in State-Space Models](https://mlsanity.com/paper/2404.08819/) (2024)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [Predictive Multiplicity in Classification](https://mlsanity.com/paper/1909.06677/) (2019)
- [A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms](https://mlsanity.com/paper/1901.10912/) (2019)
- [Taken out of context: On measuring situational awareness in LLMs](https://mlsanity.com/paper/2309.00667/) (2023)
- [Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color](https://mlsanity.com/paper/2109.06129/) (2021)
- [Discovering physical concepts with neural networks](https://mlsanity.com/paper/1807.10300/) (2018)
- [Evaluating the World Model Implicit in a Generative Model](https://mlsanity.com/paper/2406.03689/) (2024)
- [Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics](https://mlsanity.com/paper/2410.21272/) (2024)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [AI Poincar'e: Machine Learning Conservation Laws from Trajectories](https://mlsanity.com/paper/2011.04698/) (2020)
- [Chess as a Testbed for Language Model State Tracking](https://mlsanity.com/paper/2102.13249/) (2021)

## Built on by (most cited first, 30 of 47)

- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [Next-Latent Prediction Transformers Learn Compact World Models](https://mlsanity.com/paper/2511.05963/) (2025)
- [The wall confronting large language models](https://mlsanity.com/paper/2507.19703/) (2025)
- [SkyDreamer: Interpretable End-to-End Vision-Based Drone Racing with Model-Based Reinforcement Learning](https://mlsanity.com/paper/2510.14783/) (2025)
- [From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers](https://mlsanity.com/paper/2602.06923/) (2026)
- [When is a System Discoverable from Data? Discovery Requires Chaos](https://mlsanity.com/paper/2511.08860/) (2025)
- [Learning State-Tracking from Code Using Linear RNNs](https://mlsanity.com/paper/2602.14814/) (2026)
- [Overcoming Statistical Bias in Action-Controllable World Models](https://mlsanity.com/paper/2608.04653/) (2026)
- [From Observation to Insight: Mechanistic World Models and the Quest for Autonomous Discovery](https://mlsanity.com/paper/2607.12474/) (2026)
- [AI for Scientific Discovery is a Social Problem](https://mlsanity.com/paper/2509.06580/) (2025)
- [The Observer Effect in World Models: Invasive Adaptation Corrupts Latent Physics](https://mlsanity.com/paper/2602.12218/) (2026)
- [Dutch Books for Language Models](https://mlsanity.com/paper/2609.02797/) (2026)
- [From Prediction to Understanding: Will AI Foundation Models Transform Brain Science?](https://mlsanity.com/paper/2509.17280/) (2025)
- [The Impossibility of Inverse Permutation Learning in Transformer Models](https://mlsanity.com/paper/2509.24125/) (2025)
- [Convergent World Representations and Divergent Tasks](https://mlsanity.com/paper/2602.00533/) (2026)
- [SymTorch: Symbolic Distillation of Neural Networks](https://mlsanity.com/paper/2602.21307/) (2026)
- [Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling](https://mlsanity.com/paper/2607.04409/) (2026)
- [Adaptive Multi-Agent Reasoning via Automated Workflow Generation](https://mlsanity.com/paper/2507.14393/) (2025)
- [AI Needs Physics More Than Physics Needs AI](https://mlsanity.com/paper/2512.16344/) (2025)
- [Beyond Myopic World Models: Long-Horizon End-to-End Training for Direct Future Prediction](https://mlsanity.com/paper/2608.07420/) (2026)
- [PermuFormer: Multi-Task Pretraining for Permutation Representation in Algebraic Combinatorics](https://mlsanity.com/paper/2609.25438/) (2026)
- [MultiEcho: An Experimental Science of Learned Worlds](https://mlsanity.com/paper/2609.33347/) (2026)
- [Automated discovery of finite volume schemes using Graph Neural Networks](https://mlsanity.com/paper/2508.19052/) (2025)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)
- [Linguists should learn to love speech-based deep learning models](https://mlsanity.com/paper/2512.14506/) (2025)
- [Are Language Models Models?](https://mlsanity.com/paper/2601.10421/) (2026)
- [CHLU: The Causal Hamiltonian Learning Unit as a Symplectic Primitive for Deep Learning](https://mlsanity.com/paper/2603.01768/) (2026)
- [Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities](https://mlsanity.com/paper/2604.09466/) (2026)

## Abstract (arXiv)

Foundation models are premised on the idea that sequence prediction can uncover deeper domain understanding, much like how Kepler's predictions of planetary motion later led to the discovery of Newtonian mechanics. However, evaluating whether these models truly capture deeper structure remains a challenge. We develop a technique for evaluating foundation models that examines how they adapt to synthetic datasets generated from some postulated world model. Our technique measures whether the foundation model's inductive bias aligns with the world model, and so we refer to it as an inductive bias probe. Across multiple domains, we find that foundation models can excel at their training tasks yet fail to develop inductive biases towards the underlying world model when adapted to new tasks. We particularly find that foundation models trained on orbital trajectories consistently fail to apply Newtonian mechanics when adapted to new physics tasks. Further analysis reveals that these models behave as if they develop task-specific heuristics that fail to generalize.