# Exploiting Multi-Object Relationships for Detecting Adversarial Attacks in Complex Scenes

arXiv: https://arxiv.org/abs/2108.08421 · HTML: https://mlsanity.com/paper/2108.08421/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 47 references that are in the map.
- First posted on arXiv 19 Aug 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Detecting Adversarial Samples from Artifacts](https://mlsanity.com/paper/1703.00410/) (2017)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [Physical Adversarial Examples for Object Detectors](https://mlsanity.com/paper/1807.07769/) (2018)
- [ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detector](https://mlsanity.com/paper/1804.05810/) (2018)
- [Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Driving](https://mlsanity.com/paper/1907.06826/) (2019)
- [Modeling Visual Context is Key to Augmenting Object Detection Datasets](https://mlsanity.com/paper/1807.07428/) (2018)
- [Detecting Adversarial Image Examples in Deep Networks with Adaptive Noise Reduction](https://mlsanity.com/paper/1705.08378/) (2017)
- [Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships](https://mlsanity.com/paper/1807.00119/) (2018)
- [Characterizing Adversarial Examples Based on Spatial Consistency Information for Semantic Segmentation](https://mlsanity.com/paper/1810.05162/) (2018)
- [Early Methods for Detecting Adversarial Images](https://mlsanity.com/paper/1608.00530/) (2016)
- [Detection based Defense against Adversarial Examples from the Steganalysis Point of View](https://mlsanity.com/paper/1806.09186/) (2018)
- [Connecting the Dots: Detecting Adversarial Perturbations Using Context Inconsistency](https://mlsanity.com/paper/2007.09763/) (2020)

## Built on by (most cited first, 12 of 12)

- [Adversarial Attacks on Black Box Video Classifiers: Leveraging the Power of Geometric Transformations](https://mlsanity.com/paper/2110.01823/) (2021)
- [Context-Aware Transfer Attacks for Object Detection](https://mlsanity.com/paper/2112.03223/) (2021)
- [Zero-Query Transfer Attacks on Context-Aware Object Detectors](https://mlsanity.com/paper/2203.15230/) (2022)
- [GAMA: Generative Adversarial Multi-Object Scene Attacks](https://mlsanity.com/paper/2209.09502/) (2022)
- [Multi-Expert Adversarial Attack Detection in Person Re-identification Using Context Inconsistency](https://mlsanity.com/paper/2108.09891/) (2021)
- [ADC: Adversarial attacks against object Detection that evade Context consistency checks](https://mlsanity.com/paper/2110.12321/) (2021)
- [Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning](https://mlsanity.com/paper/2403.07078/) (2024)
- [Factor Graph-based Interpretable Neural Networks](https://mlsanity.com/paper/2502.14572/) (2025)
- [GLOW: Global Layout Aware Attacks on Object Detection](https://mlsanity.com/paper/2302.14166/) (2023)
- [On Feasibility of Intent Obfuscating Attacks](https://mlsanity.com/paper/2408.02674/) (2024)
- [Adversarial Attack and Disturbance Detection by Hadamard-Coded Output Representations for Object Detection and Semantic Segmentation](https://mlsanity.com/paper/2606.09536/) (2026)
- [AEGIS: A Semantic GAN and Evidential Learning Frameworkfor Robust Adversarial Detection in Vision Sensors](https://mlsanity.com/paper/2606.28416/) (2026)

## Abstract (arXiv)

Vision systems that deploy Deep Neural Networks (DNNs) are known to be vulnerable to adversarial examples. Recent research has shown that checking the intrinsic consistencies in the input data is a promising way to detect adversarial attacks (e.g., by checking the object co-occurrence relationships in complex scenes). However, existing approaches are tied to specific models and do not offer generalizability. Motivated by the observation that language descriptions of natural scene images have already captured the object co-occurrence relationships that can be learned by a language model, we develop a novel approach to perform context consistency checks using such language models. The distinguishing aspect of our approach is that it is independent of the deployed object detector and yet offers very high accuracy in terms of detecting adversarial examples in practical scenes with multiple objects.