# On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models

arXiv: https://arxiv.org/abs/1511.09249 · HTML: https://mlsanity.com/paper/1511.09249/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 1 of its 307 references that are in the map.
- First posted on arXiv 30 Nov 2015.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 1 of 1)

- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)

## Built on by (most cited first, 30 of 55)

- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [World Models for Autonomous Driving: An Initial Survey](https://mlsanity.com/paper/2403.02622/) (2024)
- [The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control](https://mlsanity.com/paper/2412.03568/) (2024)
- [Learning World Models with Identifiable Factorization](https://mlsanity.com/paper/2306.06561/) (2023)
- [Self-Improvements in Modern Agentic Systems: A Survey](https://mlsanity.com/paper/2607.13104/) (2026)
- [How to Correctly do Semantic Backpropagation on Language-based Agentic Systems](https://mlsanity.com/paper/2412.03624/) (2024)
- [The Benefits of Model-Based Generalization in Reinforcement Learning](https://mlsanity.com/paper/2211.02222/) (2022)
- [Beyond Outlining: Heterogeneous Recursive Planning for Adaptive Long-form Writing with Language Models](https://mlsanity.com/paper/2503.08275/) (2025)
- [Learning to Predict Without Looking Ahead: World Models Without Forward Prediction](https://mlsanity.com/paper/1910.13038/) (2019)
- [Learning Useful Representations of Recurrent Neural Network Weight Matrices](https://mlsanity.com/paper/2403.11998/) (2024)
- [A Step Toward World Models: A Survey on Robotic Manipulation](https://mlsanity.com/paper/2511.02097/) (2025)
- [Vidarc: Embodied Video Diffusion Model for Closed-loop Control](https://mlsanity.com/paper/2512.17661/) (2025)
- [A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures](https://mlsanity.com/paper/2602.03604/) (2026)
- [Reinforcement Learning from Delayed Observations via World Models](https://mlsanity.com/paper/2403.12309/) (2024)
- [WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG](https://mlsanity.com/paper/2603.23497/) (2026)
- [Language-Guided World Models: A Model-Based Approach to AI Control](https://mlsanity.com/paper/2402.01695/) (2024)
- [UniZero: Generalized and Efficient Planning with Scalable Latent World Models](https://mlsanity.com/paper/2406.10667/) (2024)
- [Guiding Online Reinforcement Learning with Action-Free Offline Pretraining](https://mlsanity.com/paper/2301.12876/) (2023)
- [Large Language Model Programs](https://mlsanity.com/paper/2305.05364/) (2023)
- [General Policy Evaluation and Improvement by Learning to Identify Few But Crucial States](https://mlsanity.com/paper/2207.01566/) (2022)
- [PI-ARS: Accelerating Evolution-Learned Visual-Locomotion with Predictive Information Representations](https://mlsanity.com/paper/2207.13224/) (2022)
- [Goal-Conditioned Generators of Deep Policies](https://mlsanity.com/paper/2207.01570/) (2022)
- [Learning to Identify Critical States for Reinforcement Learning from Videos](https://mlsanity.com/paper/2308.07795/) (2023)
- [AI Agents as Universal Task Solvers](https://mlsanity.com/paper/2510.12066/) (2025)
- [A Survey of Methods, Challenges and Perspectives in Causality](https://mlsanity.com/paper/2302.00293/) (2023)
- [Continual Learning Using World Models for Pseudo-Rehearsal](https://mlsanity.com/paper/1903.02647/) (2019)
- [Learning One Abstract Bit at a Time Through Self-Invented Experiments Encoded as Neural Networks](https://mlsanity.com/paper/2212.14374/) (2022)
- [FACTS: A Factored State-Space Framework For World Modelling](https://mlsanity.com/paper/2410.20922/) (2024)
- [WPT: World-to-Policy Transfer via Online World Model Distillation](https://mlsanity.com/paper/2511.20095/) (2025)

## Abstract (arXiv)

This paper addresses the general problem of reinforcement learning (RL) in partially observable environments. In 2013, our large RL recurrent neural networks (RNNs) learned from scratch to drive simulated cars from high-dimensional video input. However, real brains are more powerful in many ways. In particular, they learn a predictive model of their initially unknown environment, and somehow use it for abstract (e.g., hierarchical) planning and reasoning. Guided by algorithmic information theory, we describe RNN-based AIs (RNNAIs) designed to do the same. Such an RNNAI can be trained on never-ending sequences of tasks, some of them provided by the user, others invented by the RNNAI itself in a curious, playful fashion, to improve its RNN-based world model. Unlike our previous model-building RNN-based RL machines dating back to 1990, the RNNAI learns to actively query its model for abstract reasoning and planning and decision making, essentially "learning to think." The basic ideas of this report can be applied to many other cases where one RNN-like system exploits the algorithmic information content of another. They are taken from a grant proposal submitted in Fall 2014, and also explain concepts such as "mirror neurons." Experimental results will be described in separate papers.