# Learning POMDP World Models from Observations with Language-Model Priors

arXiv: https://arxiv.org/abs/2605.13740 · HTML: https://mlsanity.com/paper/2605.13740/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 13 May 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 9 of 9)

- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [Training Agents Inside of Scalable World Models](https://mlsanity.com/paper/2509.24527/) (2025)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning](https://mlsanity.com/paper/2107.12544/) (2021)
- [Variational Inference for Data-Efficient Model Learning in POMDPs](https://mlsanity.com/paper/1805.09281/) (2018)
- [Tru-POMDP: Task Planning Under Uncertainty via Tree of Hypotheses and Open-Ended POMDPs](https://mlsanity.com/paper/2506.02860/) (2025)
- [PIANIST: Learning Partially Observable World Models with LLMs for Multi-Agent Decision Making](https://mlsanity.com/paper/2411.15998/) (2024)
- [CASSANDRA: Programmatic and Probabilistic Learning and Inference for Stochastic World Modeling](https://mlsanity.com/paper/2601.18620/) (2026)

## Built on by (most cited first, 1 of 1)

- [EMPIRIC: Experiment-Driven Learning of Residual World Models for Robot Planning](https://mlsanity.com/paper/2609.35047/) (2026)

## Abstract (arXiv)

Whether navigating a building, operating a robot, or playing a game, an agent that acts effectively in an environment must first learn an internal model of how that environment works. Partially-observable Markov decision processes (POMDPs) provide a flexible modeling class for such internal world models, but learning them from observation-action trajectories alone is challenging and typically requires extensive environment interaction. We ask whether language-model priors can reduce costly interaction by leveraging prior knowledge, and introduce Pinductor (POMDP-inductor): an LLM proposes candidate POMDP models from a few observation-action trajectories and iteratively refines them to optimize a belief-based likelihood score. Despite using strictly less information, Pinductor matches the performance and sample efficiency of LLM-based POMDP learning methods that assume privileged access to the hidden state, while significantly surpassing the sample efficiency of tabular POMDP baselines. Further results show that performance scales with LLM capability and degrades gracefully as semantic information about the environment is withheld. Together, these results position language-model priors as a practical tool for sample-efficient world-model learning under partial observability, and a step toward generalist agents in real-world environments. Code is available at https://github.com/atomresearch/pinductor.