mlsanityData to 14 Oct 2026

How Hard is it to Confuse a World Model?

arXiv 2510.21232 ↗First posted 24 Oct 2025cs.LGcited by 1

On the map: World models → Latent dynamics models and model-based RL

also in 1 more
Abstract (arXiv)

In reinforcement learning (RL) theory, the concept of most confusing instances is central to establishing regret lower bounds, that is, the minimal exploration needed to solve a problem. Given a reference model and its optimal policy, a most confusing instance is the statistically closest alternative model that makes a suboptimal policy optimal. While this concept is well-studied in multi-armed bandits and ergodic tabular Markov decision processes, constructing such instances remains an open question in the general case. In this paper, we formalize this problem for neural network world models as a constrained optimization: finding a modified model that is statistically close to the reference one, while producing divergent performance between optimal and suboptimal policies. We propose an adversarial training procedure to solve this problem and conduct an empirical study across world models of varying quality. Our results suggest that the degree of achievable confusion correlates with uncertainty in the approximate model, which may inform theoretically-grounded exploration strategies for deep model-based RL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 12 of its 18 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

12 of 18 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Training Agents Inside of Scalable World Models29 Sep
  2. 2023TD-MPC2: Scalable, Robust World Models for Continuous Control25 Oct
  3. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May
  5. 2018World Models27 Mar

Other references · 7

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  4. 2019Self-Supervised Exploration via Disagreementcited by 265
  5. 2018Model-Based Active Explorationcited by 105
  6. 2025The regret lower bound for communicating Markov Decision Processescited by 4
  7. 2025The Confusing Instance Principle for Online Linear Quadratic Controlcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agentscited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization
  2. 2025Explainable Reinforcement Learning Agents Using World Models
  3. 2025RLVR-World: Training World Models with Reinforcement Learning
  4. 2023Model-based Offline Policy Optimization with Adversarial Network
  5. 2024A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learning
  6. 2022Towards Interpretable Deep Reinforcement Learning Models via Inverse Reinforcement Learning
  7. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  8. 2023Neural Network Approximation for Pessimistic Offline Reinforcement Learning