mlsanityData to 14 Oct 2026

IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

arXiv 2110.13214 ↗First posted 25 Oct 2021cs.CV cs.AI cs.CLcited by 267

Not on the map. Why?

Cited by 158 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Current visual question answering (VQA) tasks mainly consider answering human-annotated questions for natural images. However, aside from natural images, abstract diagrams with semantic richness are still understudied in visual understanding and reasoning research. In this work, we introduce a new challenge of Icon Question Answering (IconQA) with the goal of answering a question in an icon image context. We release IconQA, a large-scale dataset that consists of 107,439 questions and three sub-tasks: multi-image-choice, multi-text-choice, and filling-in-the-blank. The IconQA dataset is inspired by real-world diagram word problems that highlight the importance of abstract diagram understanding and comprehensive cognitive reasoning. Thus, IconQA requires not only perception skills like object recognition and text understanding, but also diverse cognitive reasoning skills, such as geometric reasoning, commonsense reasoning, and arithmetic reasoning. To facilitate potential IconQA models to learn semantic representations for icon images, we further release an icon dataset Icon645 which contains 645,687 colored icons on 377 classes. We conduct extensive user studies and blind experiments and reproduce a wide range of advanced VQA methods to benchmark the IconQA task. Also, we develop a strong IconQA baseline Patch-TRM that applies a pyramid cross-modal Transformer with input diagram embeddings pre-trained on the icon dataset. IconQA and Icon645 are available at https://iconqa.github.io.

Key facts. Cited by 267 papers in the map (data through 14 Oct 2026). Builds on 42 of its 76 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

42 of 76 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
30 more
  1. 2018Datasheets for Datasetscited by 1,386
  2. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  4. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  7. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  8. 2016A Diagram Is Worth A Dozen Imagescited by 804
  9. 2017Focal Loss for Dense Object Detectioncited by 752
  10. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  11. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  12. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  13. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  14. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  15. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  16. 2018Bilinear Attention Networkscited by 280
  17. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  18. 2017Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasmcited by 118
  19. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 114
  20. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  21. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  22. 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
  23. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 47
  24. 2017A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turkcited by 39
  25. 2018R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answeringcited by 23
  26. 2017Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answeringcited by 13
  27. 2018Question-Guided Hybrid Convolution for Visual Question Answeringcited by 12
  28. 2018Synthetically Trained Icon Proposals for Parsing and Summarizing Infographicscited by 6
  29. 2019Learning icons appearance similaritycited by 6
  30. 2020Iconify: Converting Photographs into Iconscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 38; 2024: 88; 2025: 83; 2026: 52.

267 papers.

Sort:

By area:

  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  9. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  11. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  12. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
38 more
  1. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  2. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  3. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  4. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  5. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  6. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  7. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  8. 2024What matters when building vision-language models?cited by 326
  9. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  10. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  11. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  12. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  13. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  14. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
  15. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  16. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  17. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  18. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  19. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 192
  20. 2024Large Language Models for Education: A Survey and Outlookcited by 188
  21. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  22. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
  23. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 174
  24. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  25. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  26. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  27. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  28. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  29. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  30. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 139
  31. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
  32. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  33. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  34. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  35. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  36. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  37. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  38. 2025Baichuan-Omni-1.5 Technical Reportcited by 111

50 most cited of 267 All 267 (JSON)