ColorBrowserAgent: Complex Long-Horizon Browser Agent with Adaptive Knowledge Evolution
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
With the advancement of vision-language models, web automation has made significant progress. However, deploying autonomous agents in real-world settings remains challenging, primarily due to site heterogeneity, where generalist models lack domain-specific priors for diverse interfaces, and long-horizon instability, characterized by the accumulation of decision drift over extended interactions. To address these challenges, we introduce ColorBrowserAgent (Complex Long-Horizon Browser Agent), a knowledge-evolving agent for robust web automation. Our approach addresses these challenges through two synergistic mechanisms: human-in-the-loop knowledge adaptation that transforms sparse human feedback into reusable domain knowledge, and knowledge-aligned progressive summarization that stabilizes long interactions through memory compression. Extensive experiments on WebArena, WebChoreArena and industrial deployment show that ColorBrowserAgent consistently outperforms strong baselines. It achieves a state-of-the-art success rate of 71.2% on WebArena and maintains 47.4% performance under zero-shot transfer setting on WebChoreArena. In commercial deployment, it improves user satisfaction by 19.3% relatively, verifying its robustness in real-world scenarios.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 31 references that are in the map. First posted on arXiv 12 Jan 2026.
Builds on
20 of 31 references.
References · 20
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
- 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
- 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
- 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
- 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
- 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
- 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
- 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 110
- 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 100
- 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
8 more
- 2025WebExplorer: Explore and Evolve for Training Long-Horizon Web Agentscited by 86
- 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 67
- 2023SteP: Stacked LLM Policies for Web Actionscited by 66
- 2017Human-in-the-loop Artificial Intelligencecited by 40
- 2025Go-Browse: Training Web Agents with Structured Explorationcited by 32
- 2024BPP-Search: Enhancing Tree of Thought Reasoning for Mathematical Modeling Problem Solvingcited by 15
- 2025WebOperator: Action-Aware Tree Search for Autonomous Agents in Web Environmentcited by 10
- 2025WebDART: Dynamic Decomposition and Re-planning for Complex Web Taskscited by 5
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
- 2026WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmarkcited by 4
- 2026OSCAR: Optimization-Steered Agentic Planning for Composed Image Retrievalcited by 1
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
- 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimizationcited by 0
- 2026Probe to Act: Elevating Browser-Use Agent via Active Visual Probingcited by 0
- 2026Probe to Act: Elevating Browser-Use Agent via Active Visual Probing27 Sep 2026
- 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization2 Sep 2026
- 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Design14 Jun 2026
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability8 May 2026
- 2026WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark13 Apr 2026
- 2026OSCAR: Optimization-Steered Agentic Planning for Composed Image Retrieval9 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2025BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions
- 2024AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
- 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
- 2026CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception
- 2025WebSight: A Vision-First Architecture for Robust Web Agents
- 2026MolmoWeb: Open Visual Web Agent and Open Data for the Open Web
- 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
- 2026Structured Interaction, Visual Localization, and Robust Execution for Complex Web Tasks: A Technical Report on the WebRetriever Challenge