# ViDDAR: Vision Language Model-Based Task-Detrimental Content Detection for Augmented Reality

arXiv: https://arxiv.org/abs/2501.12553 · HTML: https://mlsanity.com/paper/2501.12553/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment](https://mlsanity.com/paper/1612.01697/) (2016)
- [Better Zero-Shot Reasoning with Role-Play Prompting](https://mlsanity.com/paper/2308.07702/) (2023)
- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [Self-Chained Image-Language Model for Video Localization and Question Answering](https://mlsanity.com/paper/2305.06988/) (2023)
- [Large Language Models Empowered Autonomous Edge AI for Connected Intelligence](https://mlsanity.com/paper/2307.02779/) (2023)
- [Confusing Image Quality Assessment: Towards Better Augmented Reality Experience](https://mlsanity.com/paper/2204.04900/) (2022)
- [Saliency in Augmented Reality](https://mlsanity.com/paper/2204.08308/) (2022)
- [The Dark Side of Augmented Reality: Exploring Manipulative Designs in AR](https://mlsanity.com/paper/2303.02843/) (2023)

## Built on by (most cited first, 16 of 16)

- [A Survey on Efficient Vision-Language Models](https://mlsanity.com/paper/2504.09724/) (2025)
- [A Neurosymbolic Framework for Interpretable Cognitive Attack Detection in Augmented Reality](https://mlsanity.com/paper/2508.09185/) (2025)
- [Detecting Visual Information Manipulation Attacks in Augmented Reality: A Multimodal Semantic Reasoning Approach](https://mlsanity.com/paper/2507.20356/) (2025)
- [Perception Graph for Cognitive Attack Reasoning in Augmented Reality](https://mlsanity.com/paper/2509.05324/) (2025)
- [SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction](https://mlsanity.com/paper/2605.20110/) (2026)
- [When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactions](https://mlsanity.com/paper/2601.15308/) (2026)
- [User Prompting Strategies and Prompt Enhancement Methods for Open-Set Object Detection in XR Environments](https://mlsanity.com/paper/2601.23281/) (2026)
- [Extended to Reality: Prompt Injection in 3D Environments](https://mlsanity.com/paper/2602.07104/) (2026)
- [PRISM-XR: Empowering Privacy-Aware XR Collaboration with Multimodal Large Language Models](https://mlsanity.com/paper/2602.10154/) (2026)
- [Benchmarking Vision-Language Models under Contradictory Virtual Content Attacks in Augmented Reality](https://mlsanity.com/paper/2604.05510/) (2026)
- [Toward Safe, Trustworthy and Realistic Augmented Reality User Experience](https://mlsanity.com/paper/2507.23226/) (2025)
- [ProCap: Projection-Aware Captioning for Spatial Augmented Reality](https://mlsanity.com/paper/2604.00912/) (2026)
- [See No Evil: Semantic Context-Aware Privacy Risk Detection for AR](https://mlsanity.com/paper/2604.22805/) (2026)
- [fARfetch: Enabling Collocated AR-HRC in Large Visually Diverse Environments with VLM-Driven AR Content Adaptation](https://mlsanity.com/paper/2606.25162/) (2026)
- [Real-Time Visual Obstruction Detection in Surgical Augmented Reality](https://mlsanity.com/paper/2608.00232/) (2026)
- [Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality](https://mlsanity.com/paper/2610.00677/) (2026)

## Abstract (arXiv)

In Augmented Reality (AR), virtual content enhances user experience by providing additional information. However, improperly positioned or designed virtual content can be detrimental to task performance, as it can impair users' ability to accurately interpret real-world information. In this paper we examine two types of task-detrimental virtual content: obstruction attacks, in which virtual content prevents users from seeing real-world objects, and information manipulation attacks, in which virtual content interferes with users' ability to accurately interpret real-world information. We provide a mathematical framework to characterize these attacks and create a custom open-source dataset for attack evaluation. To address these attacks, we introduce ViDDAR (Vision language model-based Task-Detrimental content Detector for Augmented Reality), a comprehensive full-reference system that leverages Vision Language Models (VLMs) and advanced deep learning techniques to monitor and evaluate virtual content in AR environments, employing a user-edge-cloud architecture to balance performance with low latency. To the best of our knowledge, ViDDAR is the first system to employ VLMs for detecting task-detrimental content in AR settings. Our evaluation results demonstrate that ViDDAR effectively understands complex scenes and detects task-detrimental content, achieving up to 92.15% obstruction detection accuracy with a detection latency of 533 ms, and an 82.46% information manipulation content detection accuracy with a latency of 9.62 s.