mlsanityData to 7 Oct 2026

Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering

arXiv 2210.14558 ↗First posted 26 Oct 2022cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Despite the excellent performance of vision-language pre-trained models (VLPs) on conventional VQA task, they still suffer from two problems: First, VLPs tend to rely on language biases in datasets and fail to generalize to out-of-distribution (OOD) data. Second, they are inefficient in terms of memory footprint and computation. Although promising progress has been made in both problems, most existing works tackle them independently. To facilitate the application of VLP to VQA tasks, it is imperative to jointly study VLP compression and OOD robustness, which, however, has not yet been explored. This paper investigates whether a VLP can be compressed and debiased simultaneously by searching sparse and robust subnetworks. To this end, we systematically study the design of a training and compression pipeline to search the subnetworks, as well as the assignment of sparsity to different modality-specific modules. Our experiments involve 3 VLPs, 2 compression methods, 4 training methods, 2 datasets and a range of sparsity levels and random seeds. Our results show that there indeed exist sparse and robust subnetworks, which are competitive with the debiased full VLP and clearly outperform the debiasing SoTAs with fewer parameters on OOD datasets VQA-CP v2 and VQA-VS. The codes can be found at https://github.com/PhoebusSi/Compress-Robust-VQA.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 84 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

68 of 84 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
38 more
  1. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  6. 2019Are Sixteen Heads Really Better than One?cited by 803
  7. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  9. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  10. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  12. 2019The State of Sparsity in Deep Neural Networkscited by 471
  13. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  14. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  15. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  16. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  17. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  18. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  19. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  20. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  21. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  22. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  23. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  24. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  25. 2019RUBi: Reducing Unimodal Biases in Visual Question Answeringcited by 183
  26. 2020Counterfactual Samples Synthesizing for Robust Visual Question Answeringcited by 153
  27. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  28. 2019Towards Debiasing Fact Verification Modelscited by 143
  29. 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
  30. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  31. 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 104
  32. 2019Adversarial Robustness vs Model Compression, or Both?cited by 98
  33. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  34. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  35. 2019Model Compression with Adversarial Robustness: A Unified Optimization Frameworkcited by 71
  36. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  37. 2020MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answeringcited by 63
  38. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56

50 most cited of 66 All 66 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024DARE: Diverse Visual Question Answering with Robustness Evaluation
  2. 2024Probing Visual Language Priors in VLMs
  3. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
  4. 2026Rethinking Token Reduction for Large Vision-Language Models
  5. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
  6. 2024Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models
  7. 2022Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering
  8. 2024Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy