# On the Multi-turn Instruction Following for Conversational Web Agents

arXiv: https://arxiv.org/abs/2402.15057 · HTML: https://mlsanity.com/paper/2402.15057/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 34 of its 42 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 34)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018)
- [AdaPlanner: Adaptive Planning from Feedback with Language Models](https://mlsanity.com/paper/2305.16653/) (2023)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [Open Question Answering over Tables and Text](https://mlsanity.com/paper/2010.10439/) (2020)
- [OpenAgents: An Open Platform for Language Agents in the Wild](https://mlsanity.com/paper/2310.10634/) (2023)
- [Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations](https://mlsanity.com/paper/2308.16505/) (2023)
- [Large Language Models as Zero-Shot Conversational Recommenders](https://mlsanity.com/paper/2308.10053/) (2023)
- [EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Education](https://mlsanity.com/paper/2308.02773/) (2023)
- [Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents](https://mlsanity.com/paper/2311.00262/) (2023)
- [Lemur: Harmonizing Natural Language and Code for Language Agents](https://mlsanity.com/paper/2310.06830/) (2023)
- [Building Emotional Support Chatbots in the Era of LLMs](https://mlsanity.com/paper/2308.11584/) (2023)
- [PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance](https://mlsanity.com/paper/2210.08817/) (2022)

## Built on by (most cited first, 30 of 43)

- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [GUI Agents: A Survey](https://mlsanity.com/paper/2412.13501/) (2024)
- [Large Language Models Empowered Personalized Web Agents](https://mlsanity.com/paper/2410.17236/) (2024)
- [A Survey on (M)LLM-Based GUI Agents](https://mlsanity.com/paper/2504.13865/) (2025)
- [Ask-before-Plan: Proactive Language Agents for Real-World Planning](https://mlsanity.com/paper/2406.12639/) (2024)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalization](https://mlsanity.com/paper/2503.02450/) (2025)
- [OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use](https://mlsanity.com/paper/2508.04482/) (2025)
- [A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval](https://mlsanity.com/paper/2503.05659/) (2025)
- [LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects](https://mlsanity.com/paper/2504.19838/) (2025)
- [UserRL: Training Interactive User-Centric Agent via Reinforcement Learning](https://mlsanity.com/paper/2509.19736/) (2025)
- [Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents](https://mlsanity.com/paper/2602.13379/) (2026)
- [Graph-based Agent Memory: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2602.05665/) (2026)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attacks](https://mlsanity.com/paper/2502.13053/) (2025)
- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use](https://mlsanity.com/paper/2410.16400/) (2024)
- [SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agents](https://mlsanity.com/paper/2502.15538/) (2025)
- [ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agents](https://mlsanity.com/paper/2411.00927/) (2024)
- [MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models](https://mlsanity.com/paper/2505.07591/) (2025)
- [Self-Evolving World Models for LLM Agent Planning](https://mlsanity.com/paper/2606.30639/) (2026)
- [Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization](https://mlsanity.com/paper/2502.14496/) (2025)
- [UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience](https://mlsanity.com/paper/2603.24533/) (2026)
- [Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Questions](https://mlsanity.com/paper/2503.24180/) (2025)
- [RECAP: REwriting Conversations for Intent Understanding in Agentic Planning](https://mlsanity.com/paper/2509.04472/) (2025)
- [VisEscape: A Benchmark for Evaluating Exploration-driven Decision-making in Virtual Escape Rooms](https://mlsanity.com/paper/2503.14427/) (2025)
- [Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention Rewards](https://mlsanity.com/paper/2510.18731/) (2025)

## Abstract (arXiv)

Web agents powered by Large Language Models (LLMs) have demonstrated remarkable abilities in planning and executing multi-step interactions within complex web-based environments, fulfilling a wide range of web navigation tasks. Despite these advancements, the potential for LLM-powered agents to effectively engage with sequential user instructions in real-world scenarios has not been fully explored. In this work, we introduce a new task of Conversational Web Navigation, which necessitates sophisticated interactions that span multiple turns with both the users and the environment, supported by a specially developed dataset named Multi-Turn Mind2Web (MT-Mind2Web). To tackle the limited context length of LLMs and the context-dependency issue of the conversational tasks, we further propose a novel framework, named self-reflective memory-augmented planning (Self-MAP), which employs memory utilization and self-reflection techniques. Extensive experiments are conducted to benchmark the MT-Mind2Web dataset, and validate the effectiveness of the proposed method.