mlsanityData to 14 Oct 2026

EPO: Hierarchical LLM Agents with Environment Preference Optimization

arXiv 2408.16090 ↗First posted 28 Aug 2024cs.LGcited by 28

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Long-horizon decision-making tasks present significant challenges for LLM-based agents due to the need for extensive planning over multiple steps. In this paper, we propose a hierarchical framework that decomposes complex tasks into manageable subgoals, utilizing separate LLMs for subgoal prediction and low-level action generation. To address the challenge of creating training signals for unannotated datasets, we develop a reward model that leverages multimodal environment feedback to automatically generate reward signals. We introduce Environment Preference Optimization (EPO), a novel method that generates preference signals from the environment's feedback and uses them to train LLM-based agents. Extensive experiments on ALFRED demonstrate the state-of-the-art performance of our framework, achieving first place on the ALFRED public leaderboard and showcasing its potential to improve long-horizon decision-making in diverse environments.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 44 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

30 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Mask R-CNNcited by 7,277
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  9. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  11. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  12. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
17 more
  1. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  3. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  4. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  5. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  6. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
  7. 2021Episodic Transformer for Vision-and-Language Navigationcited by 156
  8. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  9. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  10. 2021FILM: Following Instructions in Language with Modular Methodscited by 121
  11. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
  12. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  13. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 78
  14. 2023Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Roboticscited by 36
  15. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33
  16. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  17. 2024Language-guided Skill Learning with Temporal Variational Inferencecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 9.

28 papers.

Sort:

By area:

  1. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
  2. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  3. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  4. 2025MPO: Boosting LLM Agents with Meta Plan Optimizationcited by 36
  5. 2025A Survey of Direct Preference Optimizationcited by 36
  6. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  7. 2025TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Reviewcited by 31
  8. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  9. 2025Coarse-to-Fine Grounded Memory for LLM Agent Planningcited by 18
  10. 2025Synthetic Video Enhances Physical Fidelity in Video Synthesiscited by 15
  11. 2024Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMscited by 9
  12. 2025World-aware Planning Narratives Enhance Large Vision-Language Model Plannercited by 9
16 more
  1. 2026Quantifying Trust: Financial Risk Management for Trustworthy AI Agentscited by 8
  2. 2025HierSearch: A Hierarchical Enterprise Deep Search Framework Integrating Local and Web Searchescited by 7
  3. 2025MADRA: Multi-Agent Debate for Risk-Aware Embodied Planningcited by 6
  4. 2025Advancing Autonomous VLM Agents via Variational Subgoal-Conditioned Reinforcement Learningcited by 5
  5. 2025Reflex First, Reflect Later: Latency-Aware Embodied LLM Agents for Dynamic Responsecited by 4
  6. 2026When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Executioncited by 3
  7. 2026HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learningcited by 3
  8. 2026Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agentscited by 2
  9. 2025How well can LLMs provide planning feedback in grounded environments?cited by 1
  10. 2026ESCAPE: Episodic Spatial Memory and Adaptive Execution Policy for Long-Horizon Mobile Manipulationcited by 1
  11. 2026ReCAPA: Hierarchical Predictive Correction to Mitigate Cascading Failurescited by 1
  12. 2024An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problemscited by 0
  13. 2025Towards a General Framework for HTN Modeling with LLMscited by 0
  14. 2026Environmental Understanding Vision-Language Model for Embodied Agentcited by 0
  15. 2026PersonalPlan: Planning Multi-Agent Systems for Personalized Programming Learningcited by 0
  16. 2026GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
  2. 2026State2State: Environment-Derived Mid-Training for LLM Agents
  3. 2026Environment-Grounded Automated Prompt Optimization for LLM Game Agents
  4. 2025Scaling Autonomous Agents via Automatic Reward Modeling And Planning
  5. 2026Matched-Context Group Policy Optimization for LLM Agent Training
  6. 2026TAPO: Transition-Aware Policy Optimization for LLM Agents
  7. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models
  8. 2026EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents