# OMNI: Open-endedness via Models of human Notions of Interestingness

arXiv: https://arxiv.org/abs/2306.01711 · HTML: https://mlsanity.com/paper/2306.01711/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 79 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 30 of 62)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [Text2Motion: From Natural Language Instructions to Feasible Plans](https://mlsanity.com/paper/2303.12153/) (2023)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Teacher-Student Curriculum Learning](https://mlsanity.com/paper/1707.00183/) (2017)
- [Language Conditioned Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2005.07648/) (2020)

## Built on by (most cited first, 30 of 70)

- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024)
- [GenSim: Generating Robotic Simulation Tasks via Large Language Models](https://mlsanity.com/paper/2310.01361/) (2023)
- [A Survey on Large Language Model-Based Game Agents](https://mlsanity.com/paper/2404.02039/) (2024)
- [Motif: Intrinsic Motivation from Artificial Intelligence Feedback](https://mlsanity.com/paper/2310.00166/) (2023)
- [STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Proving](https://mlsanity.com/paper/2502.00212/) (2025)
- [A Self-Improving Coding Agent](https://mlsanity.com/paper/2504.15228/) (2025)
- [Quality-Diversity through AI Feedback](https://mlsanity.com/paper/2310.13032/) (2023)
- [Group-Evolving Agents: Open-Ended Self-Improvement via Experience Sharing](https://mlsanity.com/paper/2602.04837/) (2026)
- [Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models](https://mlsanity.com/paper/2412.02980/) (2024)
- [Data-driven Discovery with Large Generative Models](https://mlsanity.com/paper/2402.13610/) (2024)
- [Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models](https://mlsanity.com/paper/2405.15143/) (2024)
- [AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise](https://mlsanity.com/paper/2507.00310/) (2025)
- [SIMA 2: A Generalist Embodied Agent for Virtual Worlds](https://mlsanity.com/paper/2512.04797/) (2025)
- [DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback](https://mlsanity.com/paper/2410.06215/) (2024)
- [Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis](https://mlsanity.com/paper/2505.11581/) (2025)
- [Automating the Search for Artificial Life with Foundation Models](https://mlsanity.com/paper/2412.17799/) (2024)
- [Truly Self-Improving Agents Require Intrinsic Metacognitive Learning](https://mlsanity.com/paper/2506.05109/) (2025)
- [LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teachers](https://mlsanity.com/paper/2312.08958/) (2023)
- [Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agents](https://mlsanity.com/paper/2601.12560/) (2026)
- [Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback](https://mlsanity.com/paper/2506.04287/) (2025)
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026)
- [Automated Capability Discovery via Foundation Model Self-Exploration](https://mlsanity.com/paper/2502.07577/) (2025)
- [Bootstrapping Task Spaces for Self-Improvement](https://mlsanity.com/paper/2509.04575/) (2025)
- [Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Exploration](https://mlsanity.com/paper/2410.02511/) (2024)
- [SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models](https://mlsanity.com/paper/2503.01584/) (2025)
- [From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Models](https://mlsanity.com/paper/2407.09502/) (2024)
- [Automating Traffic Model Enhancement with AI Research Agent](https://mlsanity.com/paper/2409.16876/) (2024)
- [Intrinsically-Motivated Humans and Agents in Open-World Exploration](https://mlsanity.com/paper/2503.23631/) (2025)
- [Bayesian Statistical Modeling with Predictors from LLMs](https://mlsanity.com/paper/2406.09012/) (2024)

## Abstract (arXiv)

Open-ended algorithms aim to learn new, interesting behaviors forever. That requires a vast environment search space, but there are thus infinitely many possible tasks. Even after filtering for tasks the current agent can learn (i.e., learning progress), countless learnable yet uninteresting tasks remain (e.g., minor variations of previously learned tasks). An Achilles Heel of open-endedness research is the inability to quantify (and thus prioritize) tasks that are not just learnable, but also textitinteresting (e.g., worthwhile and novel). We propose solving this problem by textitOpen-endedness via Models of human Notions of Interestingness (OMNI). The insight is that we can utilize foundation models (FMs) as a model of interestingness (MoI), because they textitalready internalize human concepts of interestingness from training on vast amounts of human-generated data, where humans naturally write about what they find interesting or boring. We show that FM-based MoIs improve open-ended learning by focusing on tasks that are both learnable textitand interesting, outperforming baselines based on uniform task sampling or learning progress alone. This approach has the potential to dramatically advance the ability to intelligently select which tasks to focus on next (i.e., auto-curricula), and could be seen as AI selecting its own next task to learn, facilitating self-improving AI and AI-Generating Algorithms. Project website at https://www.jennyzhangzt.com/omni/