MCTS-EP: Empowering Embodied Planning with Online Preference Optimization
On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning
Abstract (arXiv)
This paper introduces MCTS-EP, an online learning framework that combines large language models (LLM) with Monte Carlo Tree Search (MCTS) for training embodied agents. MCTS-EP integrates three key components: MCTS-guided exploration for preference data collection, efficient multi-modal reasoning mechanism, and iterative training pipeline based on preference optimization. We theoretically prove that MCTS-EP achieves better performance bounds than conventional on-policy algorithms when the loss function is strongly convex, and demonstrate that it can be formulated as a search-enhanced variant of GAIL. MCTS-EP achieves state-of-the-art performace across serval benchmarks. In ALFWorld, it achieves 92% and 87% success rates for textual and visual tasks. In WebShop, it reaches an average reward of 0.81. MTCS-EP also reduces average interaction steps from from 18.7/19.5 to 10.2/9.9 steps in visual ALFWorld.Code available at: https://github.com/xuhang-2/Embodied-Agent-Planning
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 57 references that are in the map. First posted on arXiv 21 Sep 2025.
Builds on
41 of 57 references.
Closest ancestors · 3
Same direction, newest first
- 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
- 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning1 May
- 2024Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing1 Feb
Other references · 38
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2023Visual Instruction Tuningcited by 7,869
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2023Qwen Technical Reportcited by 3,344
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
26 more
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
- 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
- 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
- 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
- 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
- 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
- 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
- 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
- 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
- 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
- 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 197
- 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 88
- 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
- 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
- 2020Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructionscited by 36
- 2021Language Models are Few-Shot Butlerscited by 27
- 2024APRICOT: Active Preference Learning and Constraint-Aware Task Planning with LLMscited by 11
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Reinforced Reasoning for Embodied Planning
- 2025Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learning
- 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
- 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning
- 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning
- 2025ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learning
- 2025SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents
- 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents