# COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers

arXiv: https://arxiv.org/abs/2512.02318 · HTML: https://mlsanity.com/paper/2512.02318/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 7 of its 35 references that are in the map.
- First posted on arXiv 2 Dec 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 7 of 7)

- [Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents](https://mlsanity.com/paper/2505.24878/) (2025)
- [An Empirical Study & Evaluation of Modern CAPTCHAs](https://mlsanity.com/paper/2307.12108/) (2023)
- [Breaking reCAPTCHAv2](https://mlsanity.com/paper/2409.08831/) (2024)
- [IllusionCAPTCHA: A CAPTCHA based on Visual Illusion](https://mlsanity.com/paper/2502.05461/) (2025)
- [MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attacks](https://mlsanity.com/paper/2506.05982/) (2025)
- [Reasoning under Vision: Understanding Visual-Spatial Cognition in Vision-Language Models for CAPTCHA](https://mlsanity.com/paper/2510.06067/) (2025)
- [Hacking Google reCAPTCHA v3 using Reinforcement Learning](https://mlsanity.com/paper/1903.01003/) (2019)

## Built on by (most cited first, 5 of 5)

- [Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense](https://mlsanity.com/paper/2602.09012/) (2026)
- [Artificial Effort](https://mlsanity.com/paper/2605.23920/) (2026)
- [Breaking Darknet CAPTCHAs with general purpose LLM](https://mlsanity.com/paper/2608.28794/) (2026)
- [Invisible in Space, Visible in Time: Motion Vision CAPTCHA against GUI Agents](https://mlsanity.com/paper/2609.27461/) (2026)
- [DarKnight: Retrieval-Guided Agentic Framework for Breaking Dark Web CAPTCHA Gates](https://mlsanity.com/paper/2610.19830/) (2026)

## Abstract (arXiv)

This paper studies how multimodal large language models (MLLMs) undermine the security guarantees of visual CAPTCHA. We identify the attack surface where an adversary can cheaply automate CAPTCHA solving using off-the-shelf models. We evaluate 7 representative MLLMs on 18 real-world CAPTCHA task types, measuring single-shot accuracy, success under limited retries, end-to-end latency, and per-solve cost. We further validate our findings through a supplemental external dataset and an adaptive-attacker setting with session memory, while also analyzing the impact of task-specific prompt engineering and few-shot demonstrations on solver effectiveness. We reveal that MLLMs can reliably solve recognition-oriented and low-interaction CAPTCHA tasks at human-like cost and latency, whereas tasks requiring fine-grained localization, multi-step spatial reasoning, or cross-frame consistency remain significantly harder for current models. By examining the reasoning traces of such MLLMs, we investigate the underlying mechanisms of why models succeed/fail on specific CAPTCHA puzzles and use these insights to derive defense-oriented guidelines for selecting and strengthening CAPTCHA tasks. To validate these principles, we present a proof-of-concept by hardening a vulnerable CAPTCHA type using our guidelines. We demonstrate that incorporating fine-grained localization and implicit counting reduces the success rate of state-of-the-art MLLMs from over 95% to 0%, confirming that structural changes can effectively mitigate the threat. We conclude by emphasizing the urgent need for CAPTCHA redesign as MLLM capabilities increasingly threaten existing defenses. Code Availability (https://doi.org/10.5281/zenodo.20406852).