Rejecting Hallucinated State Targets during Planning
Not on the map. Why?
Abstract (arXiv)
In planning processes of computational decision-making agents, generative or predictive models are often used as "generators" to propose "targets" representing sets of expected or desirable states. Unfortunately, learned models inevitably hallucinate infeasible targets that can cause delusional behaviors and safety concerns. We first investigate the kinds of infeasible targets that generators can hallucinate. Then, we devise a strategy to identify and reject infeasible targets by learning a target feasibility evaluator. To ensure that the evaluator is robust and non-delusional, we adopted a design choice combining off-policy compatible learning rule, distributional architecture, and data augmentation based on hindsight relabeling. Attaching to a planning agent, the designed evaluator learns by observing the agent's interactions with the environment and the targets produced by its generator, without the need to change the agent or its generator. Our controlled experiments show significant reductions in delusional behaviors and performance improvements for various kinds of existing agents.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 55 references that are in the map. First posted on arXiv 9 Oct 2024.
Builds on
30 of 55 references.
References · 30
- 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
- 2017Hindsight Experience Replaycited by 1,096
- 2020Mastering Atari with Discrete World Modelscited by 990
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
- 2017A Distributional Perspective on Reinforcement Learningcited by 743
- 2019Model-Based Reinforcement Learning for Ataricited by 495
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
- 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
- 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
- 2018Visual Reinforcement Learning with Imagined Goalscited by 248
- 2019Planning with Goal-Conditioned Policiescited by 144
18 more
- 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
- 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
- 2022Deep Hierarchical Planning from Pixelscited by 96
- 2024Understanding Hallucinations in Diffusion Models through Mode Interpolationcited by 87
- 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 40
- 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 27
- 2021A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learningcited by 27
- 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
- 2024Estimating the Hallucination Rate of Generative AIcited by 20
- 2024Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimizationcited by 15
- 2022Goal-Space Planning with Subgoal Modelscited by 12
- 2021Diversity-based Trajectory and Goal Selection with Hindsight Experience Replaycited by 6
- 2022Temporal Abstractions-Augmented Temporally Contrastive Learning: An Alternative to the Laplacian in RLcited by 6
- 2024Reconciling Spatial and Temporal Abstractions for Goal Representationcited by 6
- 2022A Look at Value-Based Decision-Time vs. Background Planning Methods Across Different Settingscited by 5
- 2024Focus On What Matters: Separated Models For Visual-Based RL Generalizationcited by 5
- 2023Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learningcited by 4
- 2018Improvements on Hindsight Learningcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.