# SimVQA: Exploring Simulated Environments for Visual Question Answering

arXiv: https://arxiv.org/abs/2203.17219 · HTML: https://mlsanity.com/paper/2203.17219/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 70 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 30 of 51)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [Driving in the Matrix: Can Virtual Worlds Replace Human-Generated Annotations for Real World Tasks?](https://mlsanity.com/paper/1610.01983/) (2016)
- [RoboTHOR: An Open Simulation-to-Real Embodied AI Platform](https://mlsanity.com/paper/2004.06799/) (2020)
- [In Defense of Grid Features for Visual Question Answering](https://mlsanity.com/paper/2001.03615/) (2020)

## Built on by (most cited first, 8 of 8)

- [SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answering](https://mlsanity.com/paper/2204.02285/) (2022)
- [Going Beyond Nouns With Vision & Language Models Using Synthetic Data](https://mlsanity.com/paper/2303.17590/) (2023)
- [Prompt-Guided Zero-Shot Anomaly Action Recognition using Pretrained Deep Skeleton Features](https://mlsanity.com/paper/2303.15167/) (2023)
- [3D-Aware Visual Question Answering about Parts, Poses and Occlusions](https://mlsanity.com/paper/2310.17914/) (2023)
- [Toward Unsupervised Realistic Visual Question Answering](https://mlsanity.com/paper/2303.05068/) (2023)
- [Is the House Ready For Sleeptime? Generating and Evaluating Situational Queries for Embodied Question Answering](https://mlsanity.com/paper/2405.04732/) (2024)
- [Selectively Hard Negative Mining for Alleviating Gradient Vanishing in Image-Text Matching](https://mlsanity.com/paper/2303.00181/) (2023)
- [SaaF: Scene-Specific Ambiguity-Aware 3D Language Fields towards Interactive Real-World Object Retrieval](https://mlsanity.com/paper/2607.16309/) (2026)

## Abstract (arXiv)

Existing work on VQA explores data augmentation to achieve better generalization by perturbing the images in the dataset or modifying the existing questions and answers. While these methods exhibit good performance, the diversity of the questions and answers are constrained by the available image set. In this work we explore using synthetic computer-generated data to fully control the visual and language space, allowing us to provide more diverse scenarios. We quantify the effect of synthetic data in real-world VQA benchmarks and to which extent it produces results that generalize to real data. By exploiting 3D and physics simulation platforms, we provide a pipeline to generate synthetic data to expand and replace type-specific questions and answers without risking the exposure of sensitive or personal data that might be present in real images. We offer a comprehensive analysis while expanding existing hyper-realistic datasets to be used for VQA. We also propose Feature Swapping (F-SWAP) -- where we randomly switch object-level features during training to make a VQA model more domain invariant. We show that F-SWAP is effective for enhancing a currently existing VQA dataset of real images without compromising on the accuracy to answer existing questions in the dataset.