# General Intelligence Requires Rethinking Exploration

arXiv: https://arxiv.org/abs/2211.07819 · HTML: https://mlsanity.com/paper/2211.07819/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 113 of its 197 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 30 of 113)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)

## Built on by (most cited first, 20 of 20)

- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [Group-Evolving Agents: Open-Ended Self-Improvement via Experience Sharing](https://mlsanity.com/paper/2602.04837/) (2026)
- [AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks](https://mlsanity.com/paper/2306.08107/) (2023)
- [Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models](https://mlsanity.com/paper/2405.15143/) (2024)
- [A social path to human-like artificial intelligence](https://mlsanity.com/paper/2405.15815/) (2024)
- [Automating the Search for Artificial Life with Foundation Models](https://mlsanity.com/paper/2412.17799/) (2024)
- [A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents](https://mlsanity.com/paper/2311.00344/) (2023)
- [Reward-Free Curricula for Training Robust World Models](https://mlsanity.com/paper/2306.09205/) (2023)
- [RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning](https://mlsanity.com/paper/2405.19548/) (2024)
- [Bootstrapping Task Spaces for Self-Improvement](https://mlsanity.com/paper/2509.04575/) (2025)
- [MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spaces](https://mlsanity.com/paper/2502.07709/) (2025)
- [An Invitation to Deep Reinforcement Learning](https://mlsanity.com/paper/2312.08365/) (2023)
- [Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environments](https://mlsanity.com/paper/2506.06981/) (2025)
- [Safety Must Precede the Deployment of Open-Ended AI](https://mlsanity.com/paper/2502.04512/) (2025)
- [Achieving Deep Continual Learning via Evolution](https://mlsanity.com/paper/2502.06210/) (2025)
- [Error Amplification Limits ANN-to-SNN Conversion in Continuous Control](https://mlsanity.com/paper/2601.21778/) (2026)
- [Augmenting Unsupervised Reinforcement Learning with Self-Reference](https://mlsanity.com/paper/2311.09692/) (2023)
- [Games of Knightian Uncertainty as AGI testbeds](https://mlsanity.com/paper/2406.18178/) (2024)
- [Artificial Intelligence and Innovation Ecosystem: Evolutionary Developments, Challenges, and Future Directions](https://mlsanity.com/paper/2607.24589/) (2026)

## Abstract (arXiv)

We are at the cusp of a transition from "learning from data" to "learning what data to learn from" as a central focus of artificial intelligence (AI) research. While the first-order learning problem is not completely solved, large models under unified architectures, such as transformers, have shifted the learning bottleneck from how to effectively train our models to how to effectively acquire and use task-relevant data. This problem, which we frame as exploration, is a universal aspect of learning in open-ended domains, such as the real world. Although the study of exploration in AI is largely limited to the field of reinforcement learning, we argue that exploration is essential to all learning systems, including supervised learning. We propose the problem of generalized exploration to conceptually unify exploration-driven learning between supervised learning and reinforcement learning, allowing us to highlight key similarities across learning settings and open research challenges. Importantly, generalized exploration serves as a necessary objective for maintaining open-ended learning processes, which in continually learning to discover and solve new problems, provides a promising path to more general intelligence.