CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training
On the map: LLM reasoning → Multimodal reasoning
also in 2 more
Abstract (arXiv)
GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices. Despite rapid progress on general GUI tasks, CAPTCHA solving remains a major challenge. On the other hand, although specialized CAPTCHA solving pipelines exist, they cannot handle general GUI tasks. To address this gap, we introduce ReCAP: a CAPTCHA-capable native GUI agent that solves modern, interactive CAPTCHA challenges while retaining general GUI-agent performance. We first develop a dynamic CAPTCHA system spanning seven representative CAPTCHA types, designed to stress primitive and complementary capabilities for CAPTCHA solving. Then, we develop an automated data collection and curation pipeline that generates large-scale CAPTCHA interaction trajectories paired with reasoning traces. As CAPTCHA solving often requires multi-step interaction and recovery from intermediate mistakes, we further leverage failed trajectories to construct self-correction data, training agents to reflect on errors and correct their actions online. Across synthetic and real-world test sets, ReCAP substantially improves CAPTCHA-solving success over its base agents, while maintaining strong performance on general GUI-agent benchmarks.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 39 references that are in the map. First posted on arXiv 23 Mar 2026.
Builds on
21 of 39 references.
References · 21
- 2025Qwen3 Technical Reportcited by 8,195
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
- 2024On the Effects of Data Scale on UI Control Agentscited by 187
- 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
- 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
- 2024GUI Agents: A Surveycited by 101
9 more
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
- 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
- 2025Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agentscited by 14
- 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 14
- 2023An Empirical Study & Evaluation of Modern CAPTCHAscited by 11
- 2025MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attackscited by 10
- 2020Capture the Bot: Using Adversarial Examples to Improve CAPTCHA Robustness to Bot Attackscited by 8
- 2025Reasoning under Vision: Understanding Visual-Spatial Cognition in Vision-Language Models for CAPTCHAcited by 4
- 2026How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factorscited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026CaptchaMind: Training CAPTCHA Solvers via Reinforcement Learning with Explicit Reasoning Supervision
- 2025COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers
- 2026CAPTCHAs in the Agentic Era: Solvers That Learn from Every Encounter
- 2025Spatial CAPTCHA: Generatively Benchmarking Spatial Reasoning for Human-Machine Differentiation
- 2026Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense
- 2026VIPER Strike: Defeating Visual Reasoning CAPTCHAs via Structured Vision-Language Inference
- 2025CAPTURE: A Benchmark and Evaluation for LVLMs in CAPTCHA Resolving
- 2026Invisible in Space, Visible in Time: Motion Vision CAPTCHA against GUI Agents