mlsanityData to 7 Oct 2026

Cross-Modal Generative Augmentation for Visual Question Answering

arXiv 2105.04780 ↗First posted 11 May 2021cs.CV cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Data augmentation has been shown to effectively improve the performance of multimodal machine learning models. This paper introduces a generative model for data augmentation by leveraging the correlations among multiple modalities. Different from conventional data augmentation approaches that apply low-level operations with deterministic heuristics, our method learns a generator that generates samples of the target modality conditioned on observed modalities in the variational auto-encoder framework. Additionally, the proposed model is able to quantify the confidence of augmented data by its generative probability, and can be jointly optimised with a downstream task. Experiments on Visual Question Answering as downstream task demonstrate the effectiveness of the proposed generative model, which is able to improve strong UpDn-based models to achieve state-of-the-art performance.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 49 of its 60 references that are in the map. First posted on arXiv 11 May 2021.

Builds on

49 of 60 references.

References · 49

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  12. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
37 more
  1. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  2. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  3. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  4. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  5. 2017Toward Controlled Generation of Textcited by 271
  6. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  7. 2020Experience Grounds Languagecited by 241
  8. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  9. 2019RUBi: Reducing Unimodal Biases in Visual Question Answeringcited by 183
  10. 2020Counterfactual Samples Synthesizing for Robust Visual Question Answeringcited by 153
  11. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  12. 2014Neural Variational Inference and Learning in Belief Networkscited by 116
  13. 2015Order-Embeddings of Images and Languagecited by 112
  14. 2015Neural Variational Inference for Text Processingcited by 111
  15. 2019CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrievalcited by 110
  16. 2014From Captions to Visual Concepts and Backcited by 109
  17. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  18. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  19. 2017Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challengecited by 76
  20. 2018Sequence-to-Sequence Data Augmentation for Dialogue Language Understandingcited by 67
  21. 2020MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answeringcited by 63
  22. 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
  23. 2019Self-Critical Reasoning for Robust Visual Question Answeringcited by 61
  24. 2016Generating Natural Questions About an Imagecited by 56
  25. 2019Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methodscited by 55
  26. 2017Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Modelscited by 53
  27. 2020Overcoming Language Priors with Self-supervised Learning for Visual Question Answeringcited by 43
  28. 2017Visual Question Generation as Dual Task of Visual Question Answeringcited by 41
  29. 2019Information Maximizing Visual Question Generationcited by 40
  30. 2019Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generationcited by 36
  31. 2020Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEscited by 31
  32. 2016Language as a Latent Variable: Discrete Generative Models for Sentence Compressioncited by 28
  33. 2020Semantic Equivalent Adversarial Data Augmentation for Visual Question Answeringcited by 26
  34. 2017Creativity: Generating Diverse Questions using Variational Autoencoderscited by 24
  35. 2018Data Augmentation for Spoken Language Understanding via Joint Variational Generationcited by 21
  36. 2020Multimodal Research in Vision and Language: A Review of Current and Emerging Trendscited by 5
  37. 2020Contrast and Classify: Training Robust VQA Modelscited by 3

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 1; 2024: 2.

7 papers.

  1. 2023Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!cited by 19
  2. 2022Rethinking Data Augmentation for Robust Visual Question Answeringcited by 18
  3. 2022Learning Multimodal Data Augmentation in Feature Spacecited by 17
  4. 2022A Feature-space Multimodal Data Augmentation Technique for Text-video Retrievalcited by 13
  5. 2022TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generationcited by 12
  6. 2024Improving Data Augmentation for Robust Visual Question Answering with Effective Curriculum Learningcited by 2
  7. 2024Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classificationcited by 2