Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agents
Not on the map. Why?
Abstract (arXiv)
Although browser-using agents (BUAs) show promise for web tasks and automation, most BUAs terminate after executing a single instruction, failing to support users' complex, nonlinear browsing with ambiguous goals, iterative decision-making, and changing contexts. We present a human-in-the-loop (HITL) conceptual framework informed by theories of human web browsing behavior. The framework centers on an iterative loop in which the BUA proactively proposes next actions and the user steers the browsing process through feedback. It also distinguishes between exploration and exploitation actions, enabling users to control the breadth and depth of their browsing. Consequently, the framework aims to reduce users' physical and cognitive effort while preserving users' traditional browsing mental model and supporting users in achieving satisfactory outcomes. We illustrate how the framework operates with hypothetical use cases and discuss the shift from manual browsing to interaction-driven browsing. We contribute a theoretically informed conceptual framework for BUAs.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 57 of its 127 references that are in the map. First posted on arXiv 15 Sep 2025.
Builds on
57 of 127 references.
References · 57
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
- 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
- 2023Language Models can Solve Computer Taskscited by 372
- 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
38 more
- 2018Reward learning from human preferences and demonstrations in Ataricited by 316
- 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
- 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
- 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 187
- 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
- 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
- 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 159
- 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
- 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
- 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
- 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 100
- 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environmentscited by 95
- 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
- 2024Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistancecited by 87
- 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
- 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 81
- 2023Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfacescited by 72
- 2024Large Language Models Empowered Personalized Web Agentscited by 66
- 2018An Overview of Machine Teachingcited by 63
- 2025Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistantcited by 54
- 2024Need Help? Designing Proactive AI Assistants for Programmingcited by 43
- 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 43
- 2024ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Datacited by 39
- 2025CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigationcited by 36
- 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 31
- 2025Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Modelcited by 26
- 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
- 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
- 2024Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?cited by 23
- 2025RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Userscited by 21
- 2024AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrationscited by 19
- 2025Build the web for agents, not agents for the webcited by 18
- 2025WebMall -- A Multi-Shop Benchmark for Evaluating Web Agentscited by 17
- 2024From Grounding to Planning: Benchmarking Bottlenecks in Web Agentscited by 12
- 2025Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Questionscited by 8
- 2022Gesture-aware Interactive Machine Teaching with In-situ Object Annotationscited by 6
- 2020On Interactive Machine Learning and the Potential of Cognitive Feedbackcited by 5
50 most cited of 57 All 57 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.