# A Large-scale Multiple-objective Method for Black-box Attack against Object Detection

arXiv: https://arxiv.org/abs/2209.07790 · HTML: https://mlsanity.com/paper/2209.07790/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 30 of 31)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks](https://mlsanity.com/paper/1904.02884/) (2019)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [HopSkipJumpAttack: A Query-Efficient Decision-Based Attack](https://mlsanity.com/paper/1904.02144/) (2019)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Adversarial Examples for Semantic Segmentation and Object Detection](https://mlsanity.com/paper/1703.08603/) (2017)
- [Enhancing the Transferability of Adversarial Attacks through Variance Tuning](https://mlsanity.com/paper/2103.15571/) (2021)
- [DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution](https://mlsanity.com/paper/2006.02334/) (2020)
- [Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical World](https://mlsanity.com/paper/2103.01050/) (2021)
- [Transferable Adversarial Attacks for Image and Video Object Detection](https://mlsanity.com/paper/1811.12641/) (2018)
- [LAS-AT: Adversarial Training with Learnable Attack Strategy](https://mlsanity.com/paper/2203.06616/) (2022)
- [Do Wider Neural Networks Really Help Adversarial Robustness?](https://mlsanity.com/paper/2010.01279/) (2020)
- [Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detection](https://mlsanity.com/paper/2201.08970/) (2022)
- [DaST: Data-free Substitute Training for Adversarial Attacks](https://mlsanity.com/paper/2003.12703/) (2020)
- [Boosting Fast Adversarial Training with Learnable Adversarial Initialization](https://mlsanity.com/paper/2110.05007/) (2021)
- [Targeted Attack for Deep Hashing based Retrieval](https://mlsanity.com/paper/2004.07955/) (2020)

## Built on by (most cited first, 30 of 55)

- [Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt](https://mlsanity.com/paper/2406.04031/) (2024)
- [VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models](https://mlsanity.com/paper/2402.13851/) (2024)
- [Enhancing Fine-Tuning Based Backdoor Defense with Sharpness-Aware Minimization](https://mlsanity.com/paper/2304.11823/) (2023)
- [Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs](https://mlsanity.com/paper/2402.14872/) (2024)
- [Manipulating Multimodal Agents via Cross-Modal Prompt Injection](https://mlsanity.com/paper/2504.14348/) (2025)
- [Visual Adversarial Attack on Vision-Language Models for Autonomous Driving](https://mlsanity.com/paper/2411.18275/) (2024)
- [Poisoned Forgery Face: Towards Backdoor Attacks on Face Forgery Detection](https://mlsanity.com/paper/2402.11473/) (2024)
- [Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning](https://mlsanity.com/paper/2403.16257/) (2024)
- [Compromising Embodied Agents with Contextual Backdoor Attacks](https://mlsanity.com/paper/2408.02882/) (2024)
- [Effectiveness Assessment of Recent Large Vision-Language Models](https://mlsanity.com/paper/2403.04306/) (2024)
- [Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attack](https://mlsanity.com/paper/2405.16134/) (2024)
- [Towards Robust Physical-world Backdoor Attacks on Lane Detection](https://mlsanity.com/paper/2405.05553/) (2024)
- [Does Few-shot Learning Suffer from Backdoor Attacks?](https://mlsanity.com/paper/2401.01377/) (2023)
- [Interpreting Object-level Foundation Models via Visual Precision Search](https://mlsanity.com/paper/2411.16198/) (2024)
- [Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation](https://mlsanity.com/paper/2209.09841/) (2022)
- [LanEvil: Benchmarking the Robustness of Lane Detection to Environmental Illusions](https://mlsanity.com/paper/2406.00934/) (2024)
- [Ensemble-based Blackbox Attacks on Dense Prediction](https://mlsanity.com/paper/2303.14304/) (2023)
- [Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025](https://mlsanity.com/paper/2506.12430/) (2025)
- [When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Models](https://mlsanity.com/paper/2602.18739/) (2026)
- [E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models](https://mlsanity.com/paper/2503.04833/) (2025)
- [Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attacks](https://mlsanity.com/paper/2308.00958/) (2023)
- [Fast Propagation is Better: Accelerating Single-Step Adversarial Training via Sampling Subnetworks](https://mlsanity.com/paper/2310.15444/) (2023)
- [Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving](https://mlsanity.com/paper/2508.02028/) (2025)
- [3D Gaussian Splatting Driven Multi-View Robust Physical Adversarial Camouflage Generation](https://mlsanity.com/paper/2507.01367/) (2025)
- [CtrlAttack: A Unified Attack on World-Model Control in Diffusion Models](https://mlsanity.com/paper/2603.13435/) (2026)
- [WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agents](https://mlsanity.com/paper/2605.23220/) (2026)
- [Face Encryption via Frequency-Restricted Identity-Agnostic Attacks](https://mlsanity.com/paper/2308.05983/) (2023)
- [Efficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threats](https://mlsanity.com/paper/2409.19526/) (2024)
- [DOEPatch: Dynamically Optimized Ensemble Model for Adversarial Patches Generation](https://mlsanity.com/paper/2312.16907/) (2023)
- [TrapFlow: Controllable Website Fingerprinting Defense via Dynamic Backdoor Learning](https://mlsanity.com/paper/2412.11471/) (2024)

## Abstract (arXiv)

Recent studies have shown that detectors based on deep models are vulnerable to adversarial examples, even in the black-box scenario where the attacker cannot access the model information. Most existing attack methods aim to minimize the true positive rate, which often shows poor attack performance, as another sub-optimal bounding box may be detected around the attacked bounding box to be the new true positive one. To settle this challenge, we propose to minimize the true positive rate and maximize the false positive rate, which can encourage more false positive objects to block the generation of new true positive bounding boxes. It is modeled as a multi-objective optimization (MOP) problem, of which the generic algorithm can search the Pareto-optimal. However, our task has more than two million decision variables, leading to low searching efficiency. Thus, we extend the standard Genetic Algorithm with Random Subset selection and Divide-and-Conquer, called GARSDC, which significantly improves the efficiency. Moreover, to alleviate the sensitivity to population quality in generic algorithms, we generate a gradient-prior initial population, utilizing the transferability between different detectors with similar backbones. Compared with the state-of-art attack methods, GARSDC decreases by an average 12.0 in the mAP and queries by about 1000 times in extensive experiments. Our codes can be found at https://github.com/LiangSiyuan21/ GARSDC.