mlsanityData to 7 Oct 2026

Semantically Distributed Robust Optimization for Vision-and-Language Inference

arXiv 2110.07165 ↗First posted 14 Oct 2021cs.CV cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Analysis of vision-and-language models has revealed their brittleness under linguistic phenomena such as paraphrasing, negation, textual entailment, and word substitutions with synonyms or antonyms. While data augmentation techniques have been designed to mitigate against these failure modes, methods that can integrate this knowledge into the training pipeline remain under-explored. In this paper, we present SDRO, a model-agnostic method that utilizes a set linguistic transformations in a distributed robust optimization setting, along with an ensembling technique to leverage these transformations during inference. Experiments on benchmark datasets with images (NLVR2) and video (VIOLIN) demonstrate performance improvements as well as robustness to adversarial attacks. Experiments on binary VQA explore the generalizability of this method to other V&L tasks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 57 of its 81 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

57 of 81 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
38 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  5. 2018Robustness May Be at Odds with Accuracycited by 873
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  7. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  8. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  9. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  10. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  11. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  12. 2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
  13. 2018Generating Natural Language Adversarial Examplescited by 422
  14. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  15. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  16. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  17. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  18. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  19. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  20. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  21. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  22. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  23. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  24. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  25. 2019Certified Robustness to Adversarial Word Substitutionscited by 164
  26. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 162
  27. 2020Counterfactual Samples Synthesizing for Robust Visual Question Answeringcited by 153
  28. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
  29. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  30. 2018A Dual Approach to Scalable Verification of Deep Networkscited by 125
  31. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  32. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  33. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  34. 2020Learning What Makes a Difference from Counterfactual Examples and Gradient Supervisioncited by 66
  35. 2020MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answeringcited by 63
  36. 2021Human-Adversarial Visual Question Answeringcited by 57
  37. 2021Ensembling with Deep Generative Viewscited by 53
  38. 2020VQA-LOL: Visual Question Answering under the Lens of Logiccited by 48

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 2; 2024: 3; 2025: 1.

9 papers.

  1. 2024Getting it Right: Improving Spatial Consistency in Text-to-Image Modelscited by 30
  2. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedbackcited by 20
  3. 2023Models See Hallucinations: Evaluating the Factuality in Video Captioningcited by 19
  4. 2022CPL: Counterfactual Prompt Learning for Vision and Language Modelscited by 18
  5. 2023AdvST: Revisiting Data Augmentations for Single Domain Generalizationcited by 16
  6. 2022CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answeringcited by 13
  7. 2024Do Large Language Models Understand Logic or Just Mimick Context?cited by 10
  8. 2021Unsupervised Natural Language Inference Using PHL Triplet Generationcited by 6
  9. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1