mlsanityData to 14 Oct 2026

Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments

arXiv 2606.03698 ↗First posted 2 Jun 2026cs.LGcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

A central goal of large language model (LLM) research is to build agentic systems that can plan, act, and adapt through sustained interaction with dynamic environments. While recent LLM-based agents exhibit impressive contextual reasoning, their long-horizon decision-making remains fragile, often suffering from objective drift, where goals and plans drift over extended interactions. We introduce Multi2, a hierarchical multi-agent decision-making framework that explicitly decomposes agent behavior into complementary roles. A high-level agent (System 1) focuses on context-aware sub-goal generation using supervised fine-tuning (SFT), while a low-level agent (System 2) executes atomic actions through offline-to-online reinforcement learning (RL) in interactive environments. This separation enables stable long-horizon control, mitigates objective drift, and allows efficient adaptation. Across diverse interactive environments, Multi2 consistently outperforms strong agentic baselines, demonstrating improved robustness and coordination in multi-turn interaction. Beyond performance, we introduce and release three hierarchical benchmark datasets, filling a long-standing gap in training and evaluating hierarchical decision-making for LLM-based agents.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 57 of its 83 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

57 of 83 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning8 Oct
  2. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 54

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2023Mistral 7Bcited by 3,164
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  11. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  12. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
38 more
  1. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  2. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 195
  3. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 177
  4. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 125
  5. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 124
  6. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  7. 2024Make Your LLM Fully Utilize the Contextcited by 119
  8. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 108
  9. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  10. 2025Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Toolscited by 106
  11. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  12. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  13. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  14. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  15. 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilitiescited by 69
  16. 2017Online Reinforcement Learning in Stochastic Gamescited by 67
  17. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 64
  18. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  19. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  20. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  21. 2024Agent-Oriented Planning in Multi-Agent Systemscited by 36
  22. 2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignmentcited by 32
  23. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  24. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  25. 2025Bridging Offline and Online Reinforcement Learning for LLMscited by 25
  26. 2025Drift No More? Context Equilibria in Multi-Turn LLM Interactionscited by 22
  27. 2025QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Searchcited by 20
  28. 2025The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvementcited by 20
  29. 2019Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamicscited by 16
  30. 2024AD4RL: Autonomous Driving Benchmarks for Offline Reinforcement Learning with Value-based Datasetcited by 15
  31. 2025DAMA: Data- and Model-aware Alignment of Multi-modal LLMscited by 14
  32. 2025ReCAP: Recursive Context-Aware Reasoning and Planning for Large Language Model Agentscited by 13
  33. 2025Data Mixing Optimization for Supervised Fine-Tuning of Large Language Modelscited by 12
  34. 2025Enhancing Decision-Making of Large Language Models via Actor-Criticcited by 7
  35. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 6
  36. 2024Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learningcited by 5
  37. 2025Path Drift in Large Reasoning Models:How First-Person Commitments Override Safetycited by 5
  38. 2025ASAP: Unsupervised Post-training with Label Distribution Shift Adaptive Learning Ratecited by 3

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agentscited by 1
  2. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution
  2. 2025A Survey on the Optimization of Large Language Model-based Agents
  3. 2025Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning
  4. 2026Multi-Agent LLMs Fail to Explore Each Other
  5. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges
  6. 2026DLM: Unified Decision Language Models for Offline Multi-Agent Sequential Decision Making
  7. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  8. 2025Controlling Performance and Budget of a Centralized Multi-agent LLM System with Reinforcement Learning