mlsanityData to 7 Oct 2026

On the Robustness of GUI Grounding Models Against Image Attacks

arXiv 2504.04716 ↗First posted 7 Apr 2025cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Graphical User Interface (GUI) grounding models are crucial for enabling intelligent agents to understand and interact with complex visual interfaces. However, these models face significant robustness challenges in real-world scenarios due to natural noise and adversarial perturbations, and their robustness remains underexplored. In this study, we systematically evaluate the robustness of state-of-the-art GUI grounding models, such as UGround, under three conditions: natural noise, untargeted adversarial attacks, and targeted adversarial attacks. Our experiments, which were conducted across a wide range of GUI environments, including mobile, desktop, and web interfaces, have clearly demonstrated that GUI grounding models exhibit a high degree of sensitivity to adversarial perturbations and low-resolution conditions. These findings provide valuable insights into the vulnerabilities of GUI grounding models and establish a strong benchmark for future research aimed at enhancing their robustness in practical applications. Our code is available at https://github.com/ZZZhr-1/Robust_GUI_Grounding.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

39 of 47 references.

References · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  9. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  10. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  11. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
27 more
  1. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  2. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  4. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  5. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  6. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  7. 2023Language Models can Solve Computer Taskscited by 371
  8. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  9. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  10. 2018Certified Defenses against Adversarial Examplescited by 305
  11. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  12. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  13. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  14. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  15. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  16. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  17. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  18. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  19. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112
  20. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attackscited by 89
  21. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Imagescited by 69
  22. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  23. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 40
  24. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
  25. 2024Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Imagecited by 26
  26. 2024Adversarial Robustness for Visual Grounding of Multimodal Large Language Modelscited by 23
  27. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agentscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 9.

14 papers.

Sort:

By area:

  1. 2025WebInject: Prompt Injection Attack to Web Agentscited by 48
  2. 2026CUA-Skill: Develop Skills for Computer Using Agentcited by 32
  3. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  4. 2025ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Searchcited by 12
  5. 2025Environmental Injection Attacks against GUI Agents in Realistic Dynamic Environmentscited by 5
  6. 2026Preference Redirection via Attention Concentration: An Attack on Computer Use Agentscited by 3
  7. 2026SnapGuard: Lightweight Prompt Injection Detection for Screenshot-Based Web Agentscited by 3
  8. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inferencecited by 2
  9. 2026AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agentscited by 2
  10. 2026Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environmentscited by 1
  11. 2026WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environmentscited by 0
  12. 2026MIRAGE: Stealthy Visual Prompt Injection for Vulnerability Detection in Web Agentscited by 0
2 more
  1. 2026MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Modelscited by 0
  2. 2026ScreenHaystack: Finding Blind Zones in GUI Groundingcited by 0