mlsanityData to 14 Oct 2026

Novelty Detection in Reinforcement Learning with World Models

arXiv 2310.08731 ↗First posted 12 Oct 2023cs.AI cs.LG cs.SYcited by 8

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Reinforcement learning (RL) using world models has found significant recent successes. However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline. We refer to the sudden change in visual properties or state transitions as novelties. Implementing novelty detection within generated world model frameworks is a crucial task for protecting the agent when deployed. In this paper, we propose straightforward bounding approaches to incorporate novelty detection into world model RL agents, by utilizing the misalignment of the world model's hallucinated states and the true observed states as an anomaly score. We provide effective approaches to detecting novelties in a distribution of transitions learned by an agent in a world model. Finally, we show the advantage of our work in a novel environment compared to traditional machine learning novelty detection methods as well as currently accepted RL focused novelty detection algorithms.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 25 of its 47 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

25 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Transformers are Sample-Efficient World Models1 Sep
  2. 2020Mastering Atari with Discrete World Models5 Oct
  3. 2018Recurrent World Models Facilitate Policy Evolution4 Sep

Other references · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  4. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  6. 2018A Survey of Unsupervised Deep Domain Adaptationcited by 361
  7. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  8. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  9. 2019Benchmarking Model-Based Reinforcement Learningcited by 162
  10. 2022Is Out-of-Distribution Detection Learnable?cited by 109
  11. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 106
  12. 2021Understanding Failures in Out-of-Distribution Detection with Deep Generative Modelscited by 89
10 more
  1. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 77
  2. 2017EX2: Exploration with Exemplar Models for Deep Reinforcement Learningcited by 42
  3. 2019On Learning Invariant Representation for Domain Adaptationcited by 39
  4. 2020Balancing reconstruction error and Kullback-Leibler divergence in Variational Autoencoderscited by 30
  5. 2023Curious Replay for Model-based Adaptationcited by 16
  6. 2021Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Resultscited by 14
  7. 2022NovGrid: A Flexible Grid World for Evaluating Agent Response to Noveltycited by 14
  8. 2020Detecting Rewards Deterioration in Episodic Reinforcement Learningcited by 8
  9. 2023Reconstruction Error-based Anomaly Detection with Few Outlying Examplescited by 5
  10. 2021Evaluating the Use of Reconstruction Error for Novelty Localizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025Continual Reinforcement Learning via Autoencoder-Driven Task and New Environment Recognitioncited by 5
  2. 2025World Model Robustness via Surprise Recognitioncited by 4
  3. 2026Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotioncited by 4
  4. 2025Shall We Play a Game? Language Models for Open-ended Wargamescited by 2
  5. 2026Self-adapting Robotic Agents through Online Continual Reinforcement Learning with World Model Feedbackcited by 2
  6. 2025Vehicle Dynamics Embedded World Models for Autonomous Drivingcited by 0
  7. 2026Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learningcited by 0
  8. 2026Changepoint-Aware World Models: Detecting Dynamics Shifts and Recovering by Forgetting Stale Replay in Model-Based RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Neuro-Symbolic World Models for Adapting to Open World Novelty
  2. 2025Bounding Distributional Shifts in World Modeling through Novelty Detection
  3. 2025World Models for Anomaly Detection during Model-Based Reinforcement Learning Inference
  4. 2025Detecting Model Drifts in Non-Stationary Environment Using Edit Operation Measures
  5. 2026Beyond Prediction: Steering VLM Agents with Retrospective World Modeling
  6. 2023Exploring the Potential of World Models for Anomaly Detection in Autonomous Driving
  7. 2025RLVR-World: Training World Models with Reinforcement Learning
  8. 2026Q-Learning With World Models