# mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning arXiv: https://arxiv.org/abs/2404.01548 ยท HTML: https://mlsanity.com/paper/2404.01548/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. - Builds on 31 of its 52 references that are in the map. - First posted on arXiv 2 Apr 2024. ## Placed in - LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/) ## Builds on (most cited first, 30 of 31) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023) - [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016) - [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023) - [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022) - [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015) - [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023) - [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023) - [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015) - [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023) - [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018) - [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022) - [Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models](https://mlsanity.com/paper/2311.06607/) (2023) - [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019) - [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017) - [PaLI-X: On Scaling up a Multilingual Vision and Language Model](https://mlsanity.com/paper/2305.18565/) (2023) - [mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding](https://mlsanity.com/paper/2307.02499/) (2023) - [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022) - [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023) - [UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model](https://mlsanity.com/paper/2310.05126/) (2023) - [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022) - [PaLI-3 Vision Language Models: Smaller, Faster, Stronger](https://mlsanity.com/paper/2310.09199/) (2023) - [ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning](https://mlsanity.com/paper/2401.02384/) (2024) - [LEAF-QA: Locate, Encode & Attend for Figure Question Answering](https://mlsanity.com/paper/1907.12861/) (2019) - [MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records](https://mlsanity.com/paper/2102.02340/) (2021) - [Answering Questions about Data Visualizations using Efficient Bimodal Fusion](https://mlsanity.com/paper/1908.01801/) (2019) - [Chart Question Answering: State of the Art and Future Directions](https://mlsanity.com/paper/2205.03966/) (2022) - [ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules](https://mlsanity.com/paper/2304.02173/) (2023) - [DOMINO: A Dual-System for Multi-step Visual Language Reasoning](https://mlsanity.com/paper/2310.02804/) (2023) - [FigureNet: A Deep Learning model for Question-Answering on Scientific Plots](https://mlsanity.com/paper/1806.04655/) (2018) ## Built on by (most cited first, 9 of 9) - [Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model](https://mlsanity.com/paper/2407.07053/) (2024) - [MMTABREAL: Real-World Benchmark for Multimodal Table Understanding](https://mlsanity.com/paper/2505.21771/) (2025) - [ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generation](https://mlsanity.com/paper/2507.19492/) (2025) - [ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answering](https://mlsanity.com/paper/2505.23242/) (2025) - [MageBench: Bridging Large Multimodal Models to Agents](https://mlsanity.com/paper/2412.04531/) (2024) - [SimVecVis: A Dataset for Enhancing MLLMs in Visualization Understanding](https://mlsanity.com/paper/2506.21319/) (2025) - [Transformers Utilization in Chart Understanding: A Review of Recent Advances & Future Trends](https://mlsanity.com/paper/2410.13883/) (2024) - [Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement](https://mlsanity.com/paper/2602.10138/) (2026) - [Chart Deep Research in LVLMs via Parallel Relative Policy Optimization](https://mlsanity.com/paper/2603.06677/) (2026) ## Abstract (arXiv) In the fields of computer vision and natural language processing, multimodal chart question-answering, especially involving color, structure, and textless charts, poses significant challenges. Traditional methods, which typically involve either direct multimodal processing or a table-to-text conversion followed by language model analysis, have limitations in effectively handling these complex scenarios. This paper introduces a novel multimodal chart question-answering model, specifically designed to address these intricate tasks. Our model integrates visual and linguistic processing, overcoming the constraints of existing methods. We adopt a dual-phase training approach: the initial phase focuses on aligning image and text representations, while the subsequent phase concentrates on optimizing the model's interpretative and analytical abilities in chart-related queries. This approach has demonstrated superior performance on multiple public datasets, particularly in handling color, structure, and textless chart questions, indicating its effectiveness in complex multimodal tasks.