# When in Doubt, Think Slow: Iterative Reasoning with Latent Imagination

arXiv: https://arxiv.org/abs/2402.15283 · HTML: https://mlsanity.com/paper/2402.15283/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 33 of its 62 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 30 of 33)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Planning to Explore via Self-Supervised World Models](https://mlsanity.com/paper/2005.05960/) (2020)
- [Thinking Fast and Slow with Deep Learning and Tree Search](https://mlsanity.com/paper/1705.08439/) (2017)
- [Provably Efficient Maximum Entropy Exploration](https://mlsanity.com/paper/1812.02690/) (2018)
- [PonderNet: Learning to Ponder](https://mlsanity.com/paper/2107.05407/) (2021)
- [SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks](https://mlsanity.com/paper/2305.17390/) (2023)
- [Adaptive Neural Networks for Efficient Inference](https://mlsanity.com/paper/1702.07811/) (2017)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks](https://mlsanity.com/paper/2106.04537/) (2021)
- [Hierarchical Multiscale Recurrent Neural Networks](https://mlsanity.com/paper/1609.01704/) (2016)
- [Iterative Amortized Inference](https://mlsanity.com/paper/1807.09356/) (2018)
- [Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning](https://mlsanity.com/paper/2107.02794/) (2021)
- [Semi-Amortized Variational Autoencoders](https://mlsanity.com/paper/1802.02550/) (2018)
- [A survey on intrinsic motivation in reinforcement learning](https://mlsanity.com/paper/1908.06976/) (2019)
- [Deep active inference agents using Monte-Carlo methods](https://mlsanity.com/paper/2006.04176/) (2020)
- [On the Relationship Between Active Inference and Control as Inference](https://mlsanity.com/paper/2006.12964/) (2020)
- [Learning Iterative Reasoning through Energy Minimization](https://mlsanity.com/paper/2206.15448/) (2022)
- [Hybrid Predictive Coding: Inferring, Fast and Slow](https://mlsanity.com/paper/2204.02169/) (2022)
- [On the challenges of learning with inference networks on sparse, high-dimensional data](https://mlsanity.com/paper/1710.06085/) (2017)
- [Iterative Amortized Policy Optimization](https://mlsanity.com/paper/2010.10670/) (2020)
- [Differentiable Adaptive Computation Time for Visual Reasoning](https://mlsanity.com/paper/2004.12770/) (2020)
- [Sample as You Infer: Predictive Coding With Langevin Dynamics](https://mlsanity.com/paper/2311.13664/) (2023)
- [Information is Power: Intrinsic Control via Information Capture](https://mlsanity.com/paper/2112.03899/) (2021)
- [Control as Hybrid Inference](https://mlsanity.com/paper/2007.05838/) (2020)

## Abstract (arXiv)

In an unfamiliar setting, a model-based reinforcement learning agent can be limited by the accuracy of its world model. In this work, we present a novel, training-free approach to improving the performance of such agents separately from planning and learning. We do so by applying iterative inference at decision-time, to fine-tune the inferred agent states based on the coherence of future state representations. Our approach achieves a consistent improvement in both reconstruction accuracy and task performance when applied to visual 3D navigation tasks. We go on to show that considering more future states further improves the performance of the agent in partially-observable environments, but not in a fully-observable one. Finally, we demonstrate that agents with less training pre-evaluation benefit most from our approach.