mlsanityData to 14 Oct 2026

Augmenting Autotelic Agents with Large Language Models

arXiv 2305.12487 ↗First posted 21 May 2023cs.AI cs.CL cs.LGcited by 30

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Humans learn to master open-ended repertoires of skills by imagining and practicing their own goals. This autotelic learning process, literally the pursuit of self-generated (auto) goals (telos), becomes more and more open-ended as the goals become more diverse, abstract and creative. The resulting exploration of the space of possible skills is supported by an inter-individual exploration: goal representations are culturally evolved and transmitted across individuals, in particular using language. Current artificial agents mostly rely on predefined goal representations corresponding to goal spaces that are either bounded (e.g. list of instructions), or unbounded (e.g. the space of possible visual inputs) but are rarely endowed with the ability to reshape their goal representations, to form new abstractions or to imagine creative goals. In this paper, we introduce a language model augmented autotelic agent (LMA3) that leverages a pretrained language model (LM) to support the representation, generation and learning of diverse, abstract, human-relevant goals. The LM is used as an imperfect model of human cultural transmission; an attempt to capture aspects of humans' common-sense, intuitive physics and overall interests. Specifically, it supports three key components of the autotelic architecture: 1)~a relabeler that describes the goals achieved in the agent's trajectories, 2)~a goal generator that suggests new high-level goals along with their decomposition into subgoals the agent already masters, and 3)~reward functions for each of these goals. Without relying on any hand-coded goal representations, reward functions or curriculum, we show that LMA3 agents learn to master a large diversity of skills in a task-agnostic text-based environment.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 41 of its 60 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

41 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents3 Feb

Other references · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2023Mastering Diverse Domains through World Modelscited by 1,253
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  9. 2017Hindsight Experience Replaycited by 1,136
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
28 more
  1. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  2. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  4. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 298
  6. 2020First return, then explorecited by 266
  7. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  8. 2020Prioritized Level Replaycited by 198
  9. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  10. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  11. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  12. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 150
  13. 2023Vision-Language Models as Success Detectorscited by 142
  14. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  15. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
  16. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 106
  17. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  18. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  19. 2020Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Surveycited by 85
  20. 2019A survey on intrinsic motivation in reinforcement learningcited by 81
  21. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 79
  22. 2015Language Understanding for Text-based Games Using Deep Reinforcement Learningcited by 71
  23. 2020Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Explorationcited by 68
  24. 2023Collaborating with language models for embodied reasoningcited by 61
  25. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  26. 2020Exploration Based Language Learning for Text-Based Gamescited by 18
  27. 2023Large language models can segment narrative events similarly to humanscited by 16
  28. 2022Language and Culture Internalisation for Human-Like Autotelic AIcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 9; 2025: 10; 2026: 4.

30 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  4. 2025Self-Challenging Language Model Agentscited by 59
  5. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  6. 2023Quality-Diversity through AI Feedbackcited by 40
  7. 2023Large Language Model as a Policy Teacher for Training Reinforcement Learning Agentscited by 30
  8. 2024Simulating Human-like Daily Activities with Desire-driven Autonomycited by 30
  9. 2024Policy Improvement using Language Feedback Modelscited by 16
  10. 2024Goals as Reward-Producing Programscited by 14
  11. 2024Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspectivecited by 11
  12. 2024Collective Innovation in Groups of Large Language Modelscited by 11
18 more
  1. 2025Automated Capability Discovery via Foundation Model Self-Explorationcited by 10
  2. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learningcited by 10
  3. 2025SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Modelscited by 9
  4. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  5. 2023ACES: Generating Diverse Programming Puzzles with with Autotelic Generative Modelscited by 6
  6. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
  7. 2025LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Modelscited by 4
  8. 2024Syllabus: Portable Curricula for Reinforcement Learning Agentscited by 2
  9. 2025A Motivational Architecture for Open-Ended Learning Challenges in Robotscited by 1
  10. 2025Expedition & Expansion: Leveraging Semantic Representations for Goal-Directed Exploration in Continuous Cellular Automatacited by 1
  11. 2025How well can LLMs provide planning feedback in grounded environments?cited by 1
  12. 2025Integrating Machine Learning into Belief-Desire-Intention Agents: Current Advances and Open Challengescited by 1
  13. 2026SimSkill: A Self-Evolving LLM Agent for Skill and Knowledge Accumulation in Traffic Simulationcited by 1
  14. 2023DeepThought: An Architecture for Autonomous Self-motivated Systemscited by 0
  15. 2025Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme Onecited by 0
  16. 2026In Search of the Ingredients of Open-Endedness: Replicating Picbreeder with Large Vision-Language Modelscited by 0
  17. 2026The Tao of Agency: Autotelic AI, Embedded Agency and Dissolution of the Selfcited by 0
  18. 2026LLM-to-Flow Distillation: Teaching Natural Language Goals to Scientific Generatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Language Model Goal Selection Differs from Humans' in a Self-Directed Learning Task
  2. 2022Language Models as Agent Models
  3. 2023Learning to Model the World with Language
  4. 2025Mind the Gap: The Divergence Between Human and LLM-Generated Tasks
  5. 2024Goal Inference from Open-Ended Dialog
  6. 2023Creative Agents: Empowering Agents with Imagination for Creative Tasks
  7. 2025Multi-Agent Language Models: Advancing Cooperation, Coordination, and Adaptation
  8. 2025Technical Report: Evaluating Goal Drift in Language Model Agents