mlsanityData to 14 Oct 2026

Multimodal Web Navigation with Instruction-Finetuned Foundation Models

arXiv 2305.11854 ↗First posted 19 May 2023cs.LG cs.AI stat.MLcited by 160

Not on the map. Why?

Cited by 112 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The progress of autonomous web navigation has been hindered by the dependence on billions of exploratory interactions via online reinforcement learning, and domain-specific model designs that make it difficult to leverage generalization from rich out-of-domain data. In this work, we study data-driven offline training for web agents with vision-language foundation models. We propose an instruction-following multimodal agent, WebGUM, that observes both webpage screenshots and HTML pages and outputs web navigation actions, such as click and type. WebGUM is trained by jointly finetuning an instruction-finetuned language model and a vision encoder with temporal and local perception on a large corpus of demonstrations. We empirically demonstrate this recipe improves the agent's ability of grounded multimodal perception, HTML comprehension, and multi-step reasoning, outperforming prior works by a significant margin. On the MiniWoB, we improve over the previous best offline methods by more than 45.8%, even outperforming online-finetuned SoTA, humans, and GPT-4-based agent. On the WebShop benchmark, our 3-billion-parameter model achieves superior performance to the existing SoTA, PaLM-540B. Furthermore, WebGUM exhibits strong positive transfer to the real-world planning tasks on the Mind2Web. We also collect 347K high-quality demonstrations using our trained models, 38 times larger than prior work, and make them available to promote future research in this direction.

Key facts. Cited by 160 papers in the map (data through 14 Oct 2026). Builds on 96 of its 114 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

96 of 114 references.

References · 96

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2023GPT-4 Technical Reportcited by 18,396
  8. 2022Training language models to follow instructions with human feedbackcited by 15,971
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  11. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,320
38 more
  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  13. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  14. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  15. 2020Learning to summarize from human feedbackcited by 2,588
  16. 2022Emergent Abilities of Large Language Modelscited by 2,512
  17. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  18. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  19. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  20. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  21. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  22. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  23. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  24. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  25. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  26. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  27. 2023PaLM 2 Technical Reportcited by 1,162
  28. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  29. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  30. 2021Scaling Vision Transformerscited by 999
  31. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  32. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  33. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  34. 2022A Generalist Agentcited by 808
  35. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  36. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  37. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  38. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569

50 most cited of 96 All 96 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 54; 2025: 59; 2026: 34.

160 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  3. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  4. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  6. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  7. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  8. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  9. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  10. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
  11. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  12. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
38 more
  1. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  2. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  3. 2024OmniParser for Pure Vision Based GUI Agentcited by 198
  4. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 195
  5. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  6. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
  7. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  8. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  9. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
  10. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 174
  11. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  12. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 161
  13. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  14. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  15. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
  16. 2024Tree Search for Language Model Agentscited by 151
  17. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 146
  18. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  19. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  20. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
  21. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
  22. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 111
  23. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 102
  24. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  25. 2025Agent Learning via Early Experiencecited by 85
  26. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  27. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79
  28. 2025Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directionscited by 77
  29. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
  30. 2023LASER: LLM Agent with State-Space Exploration for Web Navigationcited by 66
  31. 2023SteP: Stacked LLM Policies for Web Actionscited by 66
  32. 2024Large Language Models Empowered Personalized Web Agentscited by 66
  33. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 62
  34. 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wildcited by 59
  35. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 55
  36. 2024WIPI: A New Web Threat for LLM-Driven Web Agentscited by 51
  37. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  38. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization
  2. 2026UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp
  3. 2026OpAgent: Operator Agent for Web Navigation
  4. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
  5. 2024AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
  6. 2024Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning
  7. 2026From Imitation to Discrimination: Progressive Curriculum Learning for Robust Web Navigation
  8. 2026MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation