# Challenges and Prospects in Vision and Language Research

arXiv: https://arxiv.org/abs/1904.09317 · HTML: https://mlsanity.com/paper/1904.09317/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2019.

## Built on by (most cited first, 24 of 24)

- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [Learning to Predict Visual Attributes in the Wild](https://mlsanity.com/paper/2106.09707/) (2021)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks](https://mlsanity.com/paper/2212.08158/) (2022)
- [SOLO: A Single Transformer for Scalable Vision-Language Modeling](https://mlsanity.com/paper/2407.06438/) (2024)
- [Relation-aware Instance Refinement for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2103.12989/) (2021)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy](https://mlsanity.com/paper/2402.07270/) (2024)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [SIESTA: Efficient Online Continual Learning with Sleep](https://mlsanity.com/paper/2303.10725/) (2023)
- [Robust Visual Question Answering: Datasets, Methods, and Future Challenges](https://mlsanity.com/paper/2307.11471/) (2023)
- [VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13116/) (2021)
- [Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventions](https://mlsanity.com/paper/2106.04484/) (2021)
- [Are Bias Mitigation Techniques for Deep Learning Effective?](https://mlsanity.com/paper/2104.00170/) (2021)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systems](https://mlsanity.com/paper/2103.03048/) (2021)
- [Unanswerable Questions about Images and Texts](https://mlsanity.com/paper/2102.06793/) (2021)
- [Every picture tells a story: Image-grounded controllable stylistic story generation](https://mlsanity.com/paper/2209.01638/) (2022)
- [TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning](https://mlsanity.com/paper/2111.10756/) (2021)
- [Vision-and-Language Pretraining](https://mlsanity.com/paper/2207.01772/) (2022)
- [The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges](https://mlsanity.com/paper/2303.02411/) (2023)
- [Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models](https://mlsanity.com/paper/2603.06828/) (2026)
- [What is Multimodality?](https://mlsanity.com/paper/2103.06304/) (2021)
- [EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering](https://mlsanity.com/paper/2206.14355/) (2022)

## Abstract (arXiv)

Language grounded image understanding tasks have often been proposed as a method for evaluating progress in artificial intelligence. Ideally, these tasks should test a plethora of capabilities that integrate computer vision, reasoning, and natural language understanding. However, rather than behaving as visual Turing tests, recent studies have demonstrated state-of-the-art systems are achieving good performance through flaws in datasets and evaluation procedures. We review the current state of affairs and outline a path forward.