# Learning from Lexical Perturbations for Consistent Visual Question Answering

arXiv: https://arxiv.org/abs/2011.13406 · HTML: https://mlsanity.com/paper/2011.13406/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2020.

## Built on by (most cited first, 7 of 7)

- [Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models](https://mlsanity.com/paper/2310.05338/) (2023)
- [Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering](https://mlsanity.com/paper/2109.06122/) (2021)
- [SimVQA: Exploring Simulated Environments for Visual Question Answering](https://mlsanity.com/paper/2203.17219/) (2022)
- [A Comprehensive Survey on Visual Question Answering Datasets and Algorithms](https://mlsanity.com/paper/2411.11150/) (2024)
- [Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering](https://mlsanity.com/paper/2209.06954/) (2022)
- [Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information](https://mlsanity.com/paper/2305.01788/) (2023)
- [Deep Neural Networks for Visual Reasoning](https://mlsanity.com/paper/2209.11990/) (2022)

## Abstract (arXiv)

Existing Visual Question Answering (VQA) models are often fragile and sensitive to input variations. In this paper, we propose a novel approach to address this issue based on modular networks, which creates two questions related by linguistic perturbations and regularizes the visual reasoning process between them to be consistent during training. We show that our framework markedly improves consistency and generalization ability, demonstrating the value of controlled linguistic perturbations as a useful and currently underutilized training and regularization tool for VQA models. We also present VQA Perturbed Pairings (VQA P2), a new, low-cost benchmark and augmentation pipeline to create controllable linguistic variations of VQA questions. Our benchmark uniquely draws from large-scale linguistic resources, avoiding human annotation effort while maintaining data quality compared to generative approaches. We benchmark existing VQA models using VQA P2 and provide robustness analysis on each type of linguistic variation.