# 3DB: A Framework for Debugging Computer Vision Models

arXiv: https://arxiv.org/abs/2106.03805 · HTML: https://mlsanity.com/paper/2106.03805/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 95 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 30 of 63)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Adversarial Patch](https://mlsanity.com/paper/1712.09665/) (2017)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)

## Built on by (most cited first, 30 of 31)

- [Datamodels: Predicting Predictions from Training Data](https://mlsanity.com/paper/2202.00622/) (2022)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [3D Common Corruptions and Data Augmentation](https://mlsanity.com/paper/2203.01441/) (2022)
- [Salient ImageNet: How to discover spurious features in Deep Learning?](https://mlsanity.com/paper/2110.04301/) (2021)
- [A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others](https://mlsanity.com/paper/2212.04825/) (2022)
- [Diagnosing and Rectifying Vision Models using Language](https://mlsanity.com/paper/2302.04269/) (2023)
- [Adaptive Testing of Computer Vision Models](https://mlsanity.com/paper/2212.02774/) (2022)
- [Missingness Bias in Model Debugging](https://mlsanity.com/paper/2204.08945/) (2022)
- [Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generation](https://mlsanity.com/paper/2302.07865/) (2023)
- [ModelDiff: A Framework for Comparing Learning Algorithms](https://mlsanity.com/paper/2211.12491/) (2022)
- [Robust Feature-Level Adversaries are Interpretability Tools](https://mlsanity.com/paper/2110.03605/) (2021)
- [When does Bias Transfer in Transfer Learning?](https://mlsanity.com/paper/2207.02842/) (2022)
- [Red Teaming Deep Neural Networks with Feature Synthesis Tools](https://mlsanity.com/paper/2302.10894/) (2023)
- [REAP: A Large-Scale Realistic Adversarial Patch Benchmark](https://mlsanity.com/paper/2212.05680/) (2022)
- [Rethinking Backdoor Attacks](https://mlsanity.com/paper/2307.10163/) (2023)
- [OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images](https://mlsanity.com/paper/2304.10266/) (2023)
- [DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanations](https://mlsanity.com/paper/2311.17833/) (2023)
- [Trapped in texture bias? A large scale comparison of deep instance segmentation](https://mlsanity.com/paper/2401.09109/) (2024)
- [PRIME: Prioritizing Interpretability in Failure Mode Extraction](https://mlsanity.com/paper/2310.00164/) (2023)
- [FOCUS: Familiar Objects in Common and Uncommon Settings](https://mlsanity.com/paper/2110.03804/) (2021)
- [Knowledge-based in silico models and dataset for the comparative evaluation of mammography AI for a range of breast characteristics, lesion conspicuities and doses](https://mlsanity.com/paper/2310.18494/) (2023)
- [Diagnostics for Deep Neural Networks with Automated Copy/Paste Attacks](https://mlsanity.com/paper/2211.10024/) (2022)
- [Towards a More Rigorous Science of Blindspot Discovery in Image Classification Models](https://mlsanity.com/paper/2207.04104/) (2022)
- [Identifying Systematic Errors in Object Detectors with the SCROD Pipeline](https://mlsanity.com/paper/2309.13489/) (2023)
- [Causal Scene BERT: Improving object detection by searching for challenging groups of data](https://mlsanity.com/paper/2202.03651/) (2022)
- [Data-Centric Debugging: mitigating model failures via targeted data collection](https://mlsanity.com/paper/2211.09859/) (2022)
- [RenderBender: A Survey on Adversarial Attacks Using Differentiable Rendering](https://mlsanity.com/paper/2411.09749/) (2024)
- [BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance](https://mlsanity.com/paper/2506.03589/) (2025)
- [Identification of Fine-grained Systematic Errors via Controlled Scene Generation](https://mlsanity.com/paper/2404.07045/) (2024)
- [SafeFix: Targeted Model Repair via Controlled Image Generation](https://mlsanity.com/paper/2508.08701/) (2025)

## Abstract (arXiv)

We introduce 3DB: an extendable, unified framework for testing and debugging vision models using photorealistic simulation. We demonstrate, through a wide range of use cases, that 3DB allows users to discover vulnerabilities in computer vision systems and gain insights into how models make decisions. 3DB captures and generalizes many robustness analyses from prior work, and enables one to study their interplay. Finally, we find that the insights generated by the system transfer to the physical world. We are releasing 3DB as a library (https://github.com/3db/3db) alongside a set of example analyses, guides, and documentation: https://3db.github.io/3db/ .