mlsanityData to 7 Oct 2026

On the Multi-turn Instruction Following for Conversational Web Agents

arXiv 2402.15057 ↗First posted 23 Feb 2024cs.CL cs.AIcited by 42

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Web agents powered by Large Language Models (LLMs) have demonstrated remarkable abilities in planning and executing multi-step interactions within complex web-based environments, fulfilling a wide range of web navigation tasks. Despite these advancements, the potential for LLM-powered agents to effectively engage with sequential user instructions in real-world scenarios has not been fully explored. In this work, we introduce a new task of Conversational Web Navigation, which necessitates sophisticated interactions that span multiple turns with both the users and the environment, supported by a specially developed dataset named Multi-Turn Mind2Web (MT-Mind2Web). To tackle the limited context length of LLMs and the context-dependency issue of the conversational tasks, we further propose a novel framework, named self-reflective memory-augmented planning (Self-MAP), which employs memory utilization and self-reflection techniques. Extensive experiments are conducted to benchmark the MT-Mind2Web dataset, and validate the effectiveness of the proposed method.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 42 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

34 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
  2. 2024GPT-4V(ision) is a Generalist Web Agent, if Grounded3 Jan
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul
  4. 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents4 Jul

Other references · 29

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  6. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  8. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  9. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  10. 2023Language Models can Solve Computer Taskscited by 371
  11. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  12. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
17 more
  1. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  2. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  3. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 193
  4. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  5. 2020Open Question Answering over Tables and Textcited by 163
  6. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  7. 2023Recommender AI Agent: Integrating Large Language Models for Interactive Recommendationscited by 130
  8. 2023Large Language Models as Zero-Shot Conversational Recommenderscited by 126
  9. 2023EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Educationcited by 80
  10. 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agentscited by 76
  11. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  12. 2023Building Emotional Support Chatbots in the Era of LLMscited by 62
  13. 2022PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Financecited by 45
  14. 2022HybriDialogue: An Information-Seeking Dialogue Dataset Grounded on Tabular and Textual Datacited by 28
  15. 2020FLIN: A Flexible Natural Language Interface for Web Navigationcited by 18
  16. 2023Context Aware Query Rewriting for Text Rankers using LLMcited by 14
  17. 2023"What's important here?": Opportunities and Challenges of Using LLMs in Retrieving Information from Web Interfacescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 22; 2026: 12.

42 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  3. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  4. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  5. 2024GUI Agents: A Surveycited by 101
  6. 2024Large Language Models Empowered Personalized Web Agentscited by 64
  7. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  8. 2024Ask-before-Plan: Proactive Language Agents for Real-World Planningcited by 59
  9. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  10. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 44
  11. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  12. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 31
30 more
  1. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  2. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 28
  3. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 27
  4. 2024A Survey of Conversational Searchcited by 22
  5. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  6. 2025Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attackscited by 21
  7. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  8. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 17
  9. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  10. 2024ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agentscited by 14
  11. 2025MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Modelscited by 13
  12. 2025Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalizationcited by 11
  13. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  14. 2025Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Questionscited by 8
  15. 2025RECAP: REwriting Conversations for Intent Understanding in Agentic Planningcited by 7
  16. 2025VisEscape: A Benchmark for Evaluating Exploration-driven Decision-making in Virtual Escape Roomscited by 6
  17. 2025Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention Rewardscited by 6
  18. 2026Self-Evolving World Models for LLM Agent Planningcited by 5
  19. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  20. 2025Universal Retrieval for Multimodal Trajectory Modelingcited by 3
  21. 2025Investigating Co-Constructive Behavior of Large Language Models in Explanation Dialoguescited by 2
  22. 2026Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance Systemcited by 2
  23. 2026Reasoning Gets Harder for LLMs Inside A Dialoguecited by 2
  24. 2024From Context to Action: Analysis of the Impact of State Representation and Context on the Generalization of Multi-Turn Web Navigation Agentscited by 1
  25. 2025ReInAgent: A Context-Aware GUI Agent Enabling Human-in-the-Loop Mobile Task Navigationcited by 1
  26. 2026Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memorycited by 1
  27. 2026See and Remember: A Multimodal Agent for Web Traversalcited by 0
  28. 2026ReContraster: Making Your Posters Stand Out with Regional Contrastcited by 0
  29. 2026WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluationcited by 0
  30. 2026What if LLMs Ate Their Words: Causal History Effects in Multi-Turn Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
  2. 2026M2: Dual-Memory Augmentation for Long-Horizon Web Agents via Trajectory Summarization and Insight Retrieval
  3. 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance
  4. 2025WebATLAS: An LLM Agent with Experience-Driven Memory and Action Simulation
  5. 2024Large Language Models Can Self-Improve At Web Agent Tasks
  6. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
  7. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents
  8. 2026Memory in the LLM Era: Modular Architectures and Strategies in a Unified Framework