Visualizing the World Model in an Agent's Value Function

arXiv 2610.14186 ↗First posted 9 Oct 2026cs.AI cs.LGcited by 0

On the map: World models → Latent dynamics models and model-based RL

also in 1 more

Deep reinforcement learning methods → Interpretable and logic-guided RL also fits: Deep RL algorithms and representations

Abstract (arXiv)

Could model-free reinforcement learning algorithms actually be learning a model of the environment? In this work, we apply counterfactual interventions which hold everything about a state fixed, besides a targeted change, to explore if the value function of a Proximal Policy Optimization agent forms a world model of the Crafter environment. Despite only being trained to optimize reward, we discover that the value function learns concepts such as the rules governing when a diamond is collectible, water and food having an impact on health, heuristics regarding inventory, and more. Finally, we discuss the safety implications of our visualizations and some shortcomings of current reinforcement learning agents that will prevent us from building truly advanced machine intelligence.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 12 of its 27 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

12 of 27 references.

Closest ancestors · 2

Same direction, newest first

2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
2018World Models27 Mar

Other references · 10

2017Proximal Policy Optimization Algorithmscited by 15,215
2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
2017Cognitive Mapping and Planning for Visual Navigationcited by 212
2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
2017Neural Map: Structured Memory for Deep Reinforcement Learningcited by 90
2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 64
2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 39
2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 14
2022Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunitiescited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way