# QLEVR: A Diagnostic Dataset for Quantificational Language and Elementary Visual Reasoning

arXiv: https://arxiv.org/abs/2205.03075 · HTML: https://mlsanity.com/paper/2205.03075/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 6 May 2022.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input](https://mlsanity.com/paper/1410.0210/) (2014)
- [Overcoming Language Priors in Visual Question Answering with Adversarial Regularization](https://mlsanity.com/paper/1810.03649/) (2018)
- [Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies](https://mlsanity.com/paper/2010.10802/) (2020)
- [Learning Conditioned Graph Structures for Interpretable Visual Question Answering](https://mlsanity.com/paper/1806.07243/) (2018)
- [TaxiNLI: Taking a Ride up the NLU Hill](https://mlsanity.com/paper/2009.14505/) (2020)
- [High-Order Attention Models for Visual Question Answering](https://mlsanity.com/paper/1711.04323/) (2017)
- [Answer Them All! Toward Universal Visual Question Answering Models](https://mlsanity.com/paper/1903.00366/) (2019)
- [CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images](https://mlsanity.com/paper/2104.05981/) (2021)
- [Visual Question Answering: which investigated applications?](https://mlsanity.com/paper/2103.02937/) (2021)
- [Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts](https://mlsanity.com/paper/1908.10285/) (2019)
- [Part & Whole Extraction: Towards A Deep Understanding of Quantitative Facts for Percentages in Text](https://mlsanity.com/paper/2110.13505/) (2021)

## Built on by (most cited first, 5 of 5)

- [A Benchmark for Compositional Visual Reasoning](https://mlsanity.com/paper/2206.05379/) (2022)
- [Neuro-Symbolic Continual Learning: Knowledge, Reasoning Shortcuts and Concept Rehearsal](https://mlsanity.com/paper/2302.01242/) (2023)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [Emergent Communication for Rules Reasoning](https://mlsanity.com/paper/2311.04474/) (2023)
- [VisualQuest: A Benchmark for Abstract Visual Reasoning in MLLMs](https://mlsanity.com/paper/2503.19936/) (2025)

## Abstract (arXiv)

Synthetic datasets have successfully been used to probe visual question-answering datasets for their reasoning abilities. CLEVR (johnson2017clevr), for example, tests a range of visual reasoning abilities. The questions in CLEVR focus on comparisons of shapes, colors, and sizes, numerical reasoning, and existence claims. This paper introduces a minimally biased, diagnostic visual question-answering dataset, QLEVR, that goes beyond existential and numerical quantification and focus on more complex quantifiers and their combinations, e.g., asking whether there are more than two red balls that are smaller than at least three blue balls in an image. We describe how the dataset was created and present a first evaluation of state-of-the-art visual question-answering models, showing that QLEVR presents a formidable challenge to our current models. Code and Dataset are available at https://github.com/zechenli03/QLEVR