mlsanityData to 7 Oct 2026

Yin and Yang: Balancing and Answering Binary Visual Questions

arXiv 1511.05099 ↗First posted 16 Nov 2015cs.CL cs.CV cs.LGcited by 109

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The complex compositional structure of language makes problems at the intersection of vision and language challenging. But language also provides a strong prior that can result in good superficial performance, without the underlying models truly understanding the visual content. This can hinder progress in pushing state of art in the computer vision aspects of multi-modal AI. In this paper, we address binary Visual Question Answering (VQA) on abstract scenes. We formulate this problem as visual verification of concepts inquired in the questions. Specifically, we convert the question to a tuple that concisely summarizes the visual concept to be detected in the image. If the concept can be found in the image, the answer to the question is "yes", and otherwise "no". Abstract scenes play two roles (1) They allow us to focus on the high-level semantics of the VQA task as opposed to the low-level recognition problems, and perhaps more importantly, (2) They provide us the modality to balance the dataset such that language priors are controlled, and the role of vision is essential. In particular, we collect fine-grained pairs of scenes for every question, such that the answer to the question is "yes" for one scene, and "no" for the other for the exact same question. Indeed, language priors alone do not perform better than chance on our balanced dataset. Moreover, our proposed approach matches the performance of a state-of-the-art VQA approach on the unbalanced dataset, and outperforms it on the balanced dataset.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 18; 2023: 20; 2024: 26; 2025: 11; 2026: 6.

109 papers.

Sort:

By area:

  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  4. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  5. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  6. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  7. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  8. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
  9. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  10. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  11. 2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48
  12. 2021On Semantic Similarity in Video Retrievalcited by 45
38 more
  1. 2024Probing Visual Language Priors in VLMscited by 44
  2. 20213D Question Answeringcited by 41
  3. 2021Searching the Search Space of Vision Transformercited by 39
  4. 2021Greedy Gradient Ensemble for Robust Visual Question Answeringcited by 34
  5. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Taskscited by 32
  6. 2021Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQAcited by 30
  7. 2024Do Vision-Language Models Really Understand Visual Language?cited by 29
  8. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 28
  9. 2021Contrastive Vision-Language Pre-training with Limited Resourcescited by 25
  10. 2024FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culturecited by 25
  11. 2022Causal Reasoning Meets Visual Representation Learning: A Prospective Studycited by 24
  12. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  13. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  14. 2023UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Modelscited by 23
  15. 2021Rissanen Data Analysis: Examining Dataset Characteristics via Description Lengthcited by 22
  16. 2021Counterfactual Samples Synthesizing and Training for Robust Visual Question Answeringcited by 22
  17. 2022Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Taskscited by 22
  18. 2024VisionArena: 230K Real World User-VLM Conversations with Preference Labelscited by 22
  19. 2021Large-Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptionscited by 21
  20. 2021Multimodal Integration of Human-Like Attention in Visual Question Answeringcited by 20
  21. 2025A Survey on Efficient Vision-Language Modelscited by 19
  22. 2021Data Efficient Masked Language Modeling for Vision and Languagecited by 18
  23. 2022Rethinking Data Augmentation for Robust Visual Question Answeringcited by 18
  24. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 18
  25. 2021Check It Again: Progressive Visual Question Answering via Visual Entailmentcited by 17
  26. 2021VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answeringcited by 16
  27. 2021SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoningcited by 16
  28. 2022CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanationscited by 16
  29. 2022Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instancescited by 16
  30. 2021Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learningcited by 15
  31. 2021Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Visioncited by 14
  32. 2022VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challengescited by 14
  33. 2024Measuring Vision-Language STEM Skills of Neural Modelscited by 14
  34. 2024Towards Foundation Models for 3D Vision: How Close Are We?cited by 13
  35. 2022Generative Bias for Robust Visual Question Answeringcited by 12
  36. 2024Fairness and Bias Mitigation in Computer Vision: A Surveycited by 12
  37. 2024Rethinking VLMs and LLMs for Image Classificationcited by 12
  38. 2021A survey on VQA_Datasets and Approachescited by 11

50 most cited of 109 All 109 (JSON)