mlsanityData to 7 Oct 2026

On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models

arXiv 1511.09249 ↗First posted 30 Nov 2015cs.AI cs.LG cs.NEcited by 52

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

This paper addresses the general problem of reinforcement learning (RL) in partially observable environments. In 2013, our large RL recurrent neural networks (RNNs) learned from scratch to drive simulated cars from high-dimensional video input. However, real brains are more powerful in many ways. In particular, they learn a predictive model of their initially unknown environment, and somehow use it for abstract (e.g., hierarchical) planning and reasoning. Guided by algorithmic information theory, we describe RNN-based AIs (RNNAIs) designed to do the same. Such an RNNAI can be trained on never-ending sequences of tasks, some of them provided by the user, others invented by the RNNAI itself in a curious, playful fashion, to improve its RNN-based world model. Unlike our previous model-building RNN-based RL machines dating back to 1990, the RNNAI learns to actively query its model for abstract reasoning and planning and decision making, essentially "learning to think." The basic ideas of this report can be applied to many other cases where one RNN-like system exploits the algorithmic information content of another. They are taken from a grant proposal submitted in Fall 2014, and also explain concepts such as "mirror neurons." Experimental results will be described in separate papers.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 1 of its 307 references that are in the map. First posted on arXiv 30 Nov 2015.

Builds on

1 of 307 references.

References · 1

  1. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 3; 2019: 2; 2020: 0; 2021: 1; 2022: 7; 2023: 9; 2024: 8; 2025: 7; 2026: 15.

52 papers.

Sort:

By area:

  1. 2018World Modelscited by 1,444
  2. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  3. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  4. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 79
  5. 2023Learning World Models with Identifiable Factorizationcited by 23
  6. 2024How to Correctly do Semantic Backpropagation on Language-based Agentic Systemscited by 18
  7. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  8. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 17
  9. 2019Learning to Predict Without Looking Ahead: World Models Without Forward Predictioncited by 15
  10. 2025Beyond Outlining: Heterogeneous Recursive Planning for Adaptive Long-form Writing with Language Modelscited by 15
  11. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  12. 2025Vidarc: Embodied Video Diffusion Model for Closed-loop Controlcited by 15
38 more
  1. 2024Learning Useful Representations of Recurrent Neural Network Weight Matricescited by 13
  2. 2024Reinforcement Learning from Delayed Observations via World Modelscited by 13
  3. 2026A Lightweight Library for Energy-Based Joint-Embedding Predictive Architecturescited by 13
  4. 2026WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPGcited by 13
  5. 2024Language-Guided World Models: A Model-Based Approach to AI Controlcited by 12
  6. 2024UniZero: Generalized and Efficient Planning with Scalable Latent World Modelscited by 12
  7. 2023Guiding Online Reinforcement Learning with Action-Free Offline Pretrainingcited by 11
  8. 2023Large Language Model Programscited by 11
  9. 2022General Policy Evaluation and Improvement by Learning to Identify Few But Crucial Statescited by 10
  10. 2022PI-ARS: Accelerating Evolution-Learned Visual-Locomotion with Predictive Information Representationscited by 10
  11. 2022Goal-Conditioned Generators of Deep Policiescited by 9
  12. 2023Learning to Identify Critical States for Reinforcement Learning from Videoscited by 9
  13. 2025AI Agents as Universal Task Solverscited by 9
  14. 2023A Survey of Methods, Challenges and Perspectives in Causalitycited by 6
  15. 2019Continual Learning Using World Models for Pseudo-Rehearsalcited by 4
  16. 2022Learning One Abstract Bit at a Time Through Self-Invented Experiments Encoded as Neural Networkscited by 4
  17. 2024FACTS: A Factored State-Space Framework For World Modellingcited by 4
  18. 2025WPT: World-to-Policy Transfer via Online World Model Distillationcited by 4
  19. 2023Multi-Modal Cognitive Maps based on Neural Networks trained on Successor Representationscited by 3
  20. 2026Learning Abstractions for Hierarchical Planning in Program-Synthesis Agentscited by 3
  21. 2026IAPO: Information-Aware Policy Optimization for Token-Efficient Reasoningcited by 3
  22. 2021A Brief Survey of Associations Between Meta-Learning and General AIcited by 2
  23. 2022The Role of Exploration for Task Transfer in Reinforcement Learningcited by 2
  24. 2025Towards Neurocognitive-Inspired Intelligence: From AI's Structural Mimicry to Human-Like Functional Cognitioncited by 2
  25. 2026RACAS: Controlling Diverse Robots With a Single Agentic Systemcited by 2
  26. 2026Efficient Morphology-Control Co-Design via Stackelberg Proximal Policy Optimizationcited by 2
  27. 2026Interestingness as an Inductive Heuristic for Future Compression Progresscited by 2
  28. 2022Learning and Understanding a Disentangled Feature Representation for Hidden Parameters in Reinforcement Learningcited by 1
  29. 2023INTAGS: Interactive Agent-Guided Simulationcited by 1
  30. 2023Physics-Driven ML-Based Modelling for Correcting Inverse Estimationcited by 1
  31. 2025Dynamic Sparsity: Challenging Common Sparsity Assumptions for Learning World Models in Robotic Reinforcement Learning Benchmarkscited by 1
  32. 2026The Mouth is Not the Brain: Bridging Energy-Based World Models and Language Generationcited by 1
  33. 2026Coupled Control, Structured Memory, and Verifiable Action in Agentic AI (SCRAT -- Stochastic Control with Retrieval and Auditable Trajectories): A Comparative Perspective from Squirrel Locomotion and Scatter-Hoardingcited by 1
  34. 2026Three-in-One World Model: Energy-Based Consistency, Prediction, and Counterfactual Inference for Marketing Interventioncited by 1
  35. 2018Learning Shared Dynamics with Meta-World Modelscited by 0
  36. 2023QR-CLIP: Introducing Explicit Open-World Knowledge for Location and Time Reasoningcited by 0
  37. 2026From Perception to Action: Spatial AI Agents and World Modelscited by 0
  38. 2026WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modelingcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024PUZZLES: A Benchmark for Neural Algorithmic Reasoning
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
  3. 2025The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning
  4. 2025HRM-Agent: Training a recurrent reasoning model in dynamic environments using reinforcement learning
  5. 2018Learning Finite State Representations of Recurrent Policy Networks
  6. 2024VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning
  7. 2023Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning
  8. 2025RLBenchNet: The Right Network for the Right Reinforcement Learning Task