mlsanityData to 7 Oct 2026

DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?

arXiv 2605.29615 ↗First posted 28 May 2026cs.CV cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Vision-language models (VLMs) have made strong progress on high-level image-text alignment, yet their ability to perceive subtle visual differences remains limited. We study this problem in rendered web interfaces, where localized visual changes are both a diagnostic test of fine-grained perception and a practical requirement for GUI agents and design tools. We introduce DiffSpot, a code-driven benchmark for open-ended spot-the-difference on web interfaces. DiffSpot constructs controlled image pairs by mutating a single CSS property of a target element in self-contained HTML, re-rendering the page, and recording the changed property, element, and mutation magnitude. A grounding gate retains only pairs whose rendered pixel difference is confined to the target element. The benchmark contains 4,400 pairs, including 3,900 has-diff pairs balanced across 13 CSS-property operators and three difficulty tiers, plus 500 no-diff pairs for hallucination control. Evaluating 13 frontier VLMs zero-shot, we find that even the best model identifies only 40.7% of true changes, with Hard-tier Recall below 23% for every model. DiffSpot further shows that difficulty is strongly property-dependent: across CSS operators, neither pixel magnitude nor CLIP distance reliably predicts Recall.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 53 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

40 of 53 references.

References · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  9. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  10. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  12. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
28 more
  1. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  2. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  3. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  4. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  5. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 321
  6. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  7. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  8. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 191
  9. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  10. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  11. 2023Describing Differences in Image Sets with Natural Languagecited by 51
  12. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  13. 2024Probing Visual Language Priors in VLMscited by 44
  14. 2023Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answeringcited by 41
  15. 2019Neural Naturalist: Generating Fine-Grained Image Comparisonscited by 36
  16. 2025VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cuescited by 33
  17. 2024VisMin: Visual Minimal-Change Understandingcited by 23
  18. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understandingcited by 18
  19. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluationcited by 17
  20. 2024OneDiff: A Generalist Model for Image Difference Captioningcited by 15
  21. 2024CCExpert: Advancing MLLM Capability in Remote Sensing Change Captioning with Difference-Aware Integration and a Foundational Datasetcited by 13
  22. 2024VIXEN: Visual Text Comparison Network for Image Difference Captioningcited by 12
  23. 2025OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioningcited by 12
  24. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 12
  25. 2025BTCChat: Advancing Remote Sensing Bi-temporal Change Captioning with Multimodal Large Language Modelcited by 7
  26. 2025Same or Not? Enhancing Visual Perception in Vision-Language Modelscited by 5
  27. 2026VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?cited by 4
  28. 2025STATUS Bench: A Rigorous Benchmark for Evaluating Object State Understanding in Vision-Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation
  2. 2025WebRenderBench: Enhancing Web Interface Generation through Layout-Style Consistency and Reinforcement Learning
  3. 2026UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
  4. 2024IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web
  5. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow
  6. 2025RWKV-UI: UI Understanding with Enhanced Perception and Reasoning
  7. 2025Visual Test-time Scaling for GUI Agent Grounding
  8. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset