mlsanityData to 7 Oct 2026

Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents

arXiv 2602.00929 ↗First posted 31 Jan 2026cs.AIcited by 3

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Humans learn abstractions and use them to plan efficiently to quickly generalize across tasks -- an ability that remains challenging for state-of-the-art large language model (LLM) agents and deep reinforcement learning (RL) systems. Inspired by the cognitive science of how people form abstractions and intuitive theories of their world knowledge, Theory-Based RL (TBRL) systems, such as TheoryCoder, exhibit strong generalization through effective use of abstractions. However, they heavily rely on human-provided abstractions and sidestep the abstraction-learning problem. We introduce TheoryCoder-2, a new TBRL agent that leverages LLMs' in-context learning ability to actively learn reusable abstractions rather than relying on hand-specified ones, by synthesizing abstractions from experience and integrating them into a hierarchical planning process. We conduct experiments on diverse environments, including BabyAI, Minihack and VGDL games like Sokoban. We find that TheoryCoder-2 is significantly more sample-efficient than baseline LLM agents augmented with classical planning domain construction, reasoning-based planning, and prior program-synthesis agents such as WorldCoder. TheoryCoder-2 is able to solve complex tasks that the baselines fail, while only requiring minimal human prompts, unlike prior TBRL systems.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 39 of its 68 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

39 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games20 Nov
  2. 2024SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling16 Oct
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  7. 2023Mastering Diverse Domains through World Modelscited by 1,193
  8. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  9. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  10. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  11. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  12. 2016The Option-Critic Architecturecited by 433
24 more
  1. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  2. 2021Mastering Atari Games with Limited Datacited by 235
  3. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  4. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  5. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  6. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 99
  7. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  8. 2025Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoningcited by 82
  9. 2015On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Modelscited by 52
  10. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  11. 2021Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planningcited by 43
  12. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  13. 2024VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planningcited by 37
  14. 2023Learning adaptive planning representations with natural language guidancecited by 36
  15. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  16. 2025PoE-World: Compositional World Modeling with Products of Programmatic Expertscited by 25
  17. 2017Learning model-based planning from scratchcited by 21
  18. 2022Humans decompose tasks by trading off utility and computational costcited by 18
  19. 2021Learning Task Decomposition with Ordered Memory Policy Networkcited by 15
  20. 2022Unsupervised Learning of Temporal Abstractions with Slot-based Transformerscited by 14
  21. 2024Generating consistent PDDL domains with Large Language Modelscited by 14
  22. 2025Synthesizing world models for bilevel planningcited by 13
  23. 2024Baba Is AI: Break the Rules to Beat the Benchmarkcited by 10
  24. 2024Interactive and Expressive Code-Augmented Planning with Large Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Tycho: Active Abstraction with Programmatic World Models for ARC-AGI-3cited by 5
  2. 2026Code to Control: Synthesizing Parameterized Reactive Controllerscited by 0
  3. 2026Schema: Discovering Unknown Environments via Agentic Program Inductioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents
  2. 2023AdaPlanner: Adaptive Planning from Feedback with Language Models
  3. 2026From Coarse to Fine: Self-Adaptive Hierarchical Planning for LLM Agents
  4. 2026Learning and Reusing Policy Decompositions for Hierarchical Generalized Planning with LLM Agents
  5. 2026Up and Down the Abstraction Ladder: Code-Based Skills for Language Agents
  6. 2025Policy-Conditioned Policies for Multi-Agent Task Solving
  7. 2025PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning
  8. 2025Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning