mlsanityData to 7 Oct 2026

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

arXiv 2402.05930 ↗First posted 8 Feb 2024cs.CL cs.CV cs.LGcited by 177

Not on the map. Why?

Cited by 137 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve real-world tasks in a multi-turn dialogue fashion. To support this problem, we introduce WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers a broad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller finetuned decoders surpass the best zero-shot LLMs (including GPT-4V), but also larger finetuned multimodal models which were explicitly pretrained on screenshots. However, all finetuned models struggle to generalize to unseen websites. Our findings highlight the need for large multimodal models that can generalize to novel settings. Our code, data and models are available for research: https://mcgill-nlp.github.io/weblinx

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Builds on 62 of its 101 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

62 of 101 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  13. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  14. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  15. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  16. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  17. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  18. 2016A Diagram Is Worth A Dozen Imagescited by 793
  19. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  20. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  21. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  22. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  23. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  24. 2023Language Models can Solve Computer Taskscited by 371
  25. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  26. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  27. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  28. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  29. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  30. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  31. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  32. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  33. 2021TopiOCQA: Open-domain Conversational Question Answering with Topic Switchingcited by 95
  34. 2022A data-driven approach for learning to control computerscited by 87
  35. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  36. 2020Recent Advances and Challenges in Task-oriented Dialog Systemcited by 78
  37. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  38. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 70; 2026: 67.

177 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  3. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  4. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  5. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  6. 2024AI Agents That Mattercited by 199
  7. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  8. 2025Survey on Evaluation of LLM-based Agentscited by 190
  9. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  10. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  11. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  12. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
38 more
  1. 2024Tree Search for Language Model Agentscited by 148
  2. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  3. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 119
  4. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  5. 2024Large Multimodal Agents: A Surveycited by 99
  6. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 99
  7. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  8. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 97
  9. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 89
  10. 2024WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Taskscited by 81
  11. 2024Beyond Browsing: API-Based Web Agentscited by 77
  12. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 69
  13. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  14. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  15. 2024OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automationcited by 66
  16. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  17. 2024Large Language Models Empowered Personalized Web Agentscited by 64
  18. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  19. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  20. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 55
  21. 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wildcited by 55
  22. 2025OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflowscited by 54
  23. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  24. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  25. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  26. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 42
  27. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 38
  28. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  29. 2025CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigationcited by 35
  30. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  31. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  32. 2025InSTA: Towards Internet-Scale Training For Agentscited by 31
  33. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 31
  34. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  35. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 28
  36. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 27
  37. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  38. 2025OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agentscited by 25

50 most cited of 177 All 177 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Multi-turn Instruction Following for Conversational Web Agents
  2. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agent
  3. 2024Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning
  4. 2025BEARCUBS: A benchmark for computer-using web agents
  5. 2023AllTogether: Investigating the Efficacy of Spliced Prompt for Web Navigation using Large Language Models
  6. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites
  7. 2025Go-Browse: Training Web Agents with Structured Exploration
  8. 2025BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks