# OpenCQA: Open-ended Question Answering with Charts

arXiv: https://arxiv.org/abs/2210.06628 · HTML: https://mlsanity.com/paper/2210.06628/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 69 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Memorizing Transformers](https://mlsanity.com/paper/2203.08913/) (2022)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)

## Built on by (most cited first, 30 of 61)

- [VITA: Towards Open-Source Interactive Omni Multimodal LLM](https://mlsanity.com/paper/2408.05211/) (2024)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning](https://mlsanity.com/paper/2402.12185/) (2024)
- [ChartBench: A Benchmark for Complex Visual Reasoning in Charts](https://mlsanity.com/paper/2312.15915/) (2023)
- [ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering](https://mlsanity.com/paper/2504.05506/) (2025)
- [ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning](https://mlsanity.com/paper/2401.02384/) (2024)
- [ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild](https://mlsanity.com/paper/2407.04172/) (2024)
- [ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2403.09028/) (2024)
- [From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models](https://mlsanity.com/paper/2403.12027/) (2024)
- [OneChart: Purify the Chart Structural Extraction via One Auxiliary Token](https://mlsanity.com/paper/2404.09987/) (2024)
- [ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries](https://mlsanity.com/paper/2304.13620/) (2023)
- [Distill Visual Chart Reasoning Ability from LLMs to MLLMs](https://mlsanity.com/paper/2410.18798/) (2024)
- [Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs](https://mlsanity.com/paper/2406.00257/) (2024)
- [DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts](https://mlsanity.com/paper/2408.05346/) (2024)
- [ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules](https://mlsanity.com/paper/2304.02173/) (2023)
- [Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering](https://mlsanity.com/paper/2410.08085/) (2024)
- [OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answering in Vietnamese](https://mlsanity.com/paper/2305.04183/) (2023)
- [Effective Training Data Synthesis for Improving MLLM Chart Understanding](https://mlsanity.com/paper/2508.06492/) (2025)
- [Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness](https://mlsanity.com/paper/2407.11229/) (2024)
- [InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Charts](https://mlsanity.com/paper/2505.19028/) (2025)
- [What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases](https://mlsanity.com/paper/2404.02415/) (2024)
- [On Pre-training of Multimodal Language Models Customized for Chart Understanding](https://mlsanity.com/paper/2407.14506/) (2024)
- [BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning](https://mlsanity.com/paper/2508.09804/) (2025)
- [Boosting Chart-to-Code Generation in MLLM via Dual Preference-Guided Refinement](https://mlsanity.com/paper/2504.02906/) (2025)
- [Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarization](https://mlsanity.com/paper/2312.10610/) (2023)
- [ChartM3: Benchmarking Chart Editing with Multimodal Instructions](https://mlsanity.com/paper/2507.21167/) (2025)
- [ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Images](https://mlsanity.com/paper/2404.18397/) (2024)
- [WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts](https://mlsanity.com/paper/2506.15594/) (2025)
- [SIMPLOT: Enhancing Chart Question Answering by Distilling Essentials](https://mlsanity.com/paper/2405.00021/) (2024)
- [ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answering](https://mlsanity.com/paper/2505.23242/) (2025)

## Abstract (arXiv)

Charts are very popular to analyze data and convey important insights. People often analyze visualizations to answer open-ended questions that require explanatory answers. Answering such questions are often difficult and time-consuming as it requires a lot of cognitive and perceptual efforts. To address this challenge, we introduce a new task called OpenCQA, where the goal is to answer an open-ended question about a chart with descriptive texts. We present the annotation process and an in-depth analysis of our dataset. We implement and evaluate a set of baselines under three practical settings. In the first setting, a chart and the accompanying article is provided as input to the model. The second setting provides only the relevant paragraph(s) to the chart instead of the entire article, whereas the third setting requires the model to generate an answer solely based on the chart. Our analysis of the results show that the top performing models generally produce fluent and coherent text while they struggle to perform complex logical and arithmetic reasoning.