# ChartCheck: Explainable Fact-Checking over Real-World Chart Images

arXiv: https://arxiv.org/abs/2311.07453 · HTML: https://mlsanity.com/paper/2311.07453/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 26 of 26)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [Explainable Automated Fact-Checking for Public Health Claims](https://mlsanity.com/paper/2010.09926/) (2020)
- [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://mlsanity.com/paper/1909.03242/) (2019)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information](https://mlsanity.com/paper/2106.05707/) (2021)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking](https://mlsanity.com/paper/2301.11843/) (2023)
- [Answering Questions about Data Visualizations using Efficient Bimodal Fusion](https://mlsanity.com/paper/1908.01801/) (2019)
- [Misinformed by Visualization: What Do We Learn From Misinformative Visualizations?](https://mlsanity.com/paper/2204.09548/) (2022)
- [Viral Visualizations: How Coronavirus Skeptics Use Orthodox Data Practices to Promote Unorthodox Science Online](https://mlsanity.com/paper/2101.07993/) (2021)
- [Why Change My Design: Explaining Poorly Constructed Visualization Designs with Explorable Explanations](https://mlsanity.com/paper/2309.01445/) (2023)

## Built on by (most cited first, 26 of 26)

- [ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning](https://mlsanity.com/paper/2402.12185/) (2024)
- [DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts](https://mlsanity.com/paper/2412.10510/) (2024)
- [Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs](https://mlsanity.com/paper/2406.00257/) (2024)
- [DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts](https://mlsanity.com/paper/2408.05346/) (2024)
- [Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking](https://mlsanity.com/paper/2411.05375/) (2024)
- [SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification](https://mlsanity.com/paper/2506.15569/) (2025)
- [BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning](https://mlsanity.com/paper/2508.09804/) (2025)
- [MuSciClaims: Multimodal Scientific Claim Verification](https://mlsanity.com/paper/2506.04585/) (2025)
- [ChartCap: Mitigating Hallucination of Dense Chart Captioning](https://mlsanity.com/paper/2508.03164/) (2025)
- [SciClaimEval: Cross-modal Claim Verification in Scientific Papers](https://mlsanity.com/paper/2602.07621/) (2026)
- [The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers](https://mlsanity.com/paper/2506.20844/) (2025)
- [Table-Text Alignment: Explaining Claim Verification Against Tables in Scientific Papers](https://mlsanity.com/paper/2506.10486/) (2025)
- [ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts](https://mlsanity.com/paper/2506.08700/) (2025)
- [TSVer: A Benchmark for Fact Verification Against Time-Series Evidence](https://mlsanity.com/paper/2511.01101/) (2025)
- [Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework](https://mlsanity.com/paper/2606.29808/) (2026)
- [M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency](https://mlsanity.com/paper/2604.01306/) (2026)
- [Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Tasks](https://mlsanity.com/paper/2510.06071/) (2025)
- [MEVER: Multi-Modal and Explainable Claim Verification with Graph-based Evidence Retrieval](https://mlsanity.com/paper/2602.10023/) (2026)
- [When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations](https://mlsanity.com/paper/2603.22368/) (2026)
- [Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity](https://mlsanity.com/paper/2604.04692/) (2026)
- [When Verification Fails: How Compositionally Infeasible Claims Escape Rejection](https://mlsanity.com/paper/2604.10990/) (2026)
- [IPO-Mine: A Toolkit and Dataset for Section-Structured Analysis of Long, Multimodal IPO Documents](https://mlsanity.com/paper/2605.28714/) (2026)
- [ElementCheck: Complexity-Aware Long-Form Text Factuality Evaluation via Sentence Elements](https://mlsanity.com/paper/2608.26118/) (2026)
- [InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations](https://mlsanity.com/paper/2609.01383/) (2026)
- [Faithful Chart Generation for Multimodal Deep Research: Frame-Evidence Co-Adaptation](https://mlsanity.com/paper/2610.00374/) (2026)
- [Do you "see" what I "see"? A Multi-panel Visual Question Answer Dataset for Large Language Model Chart Analysis](https://mlsanity.com/paper/2610.14173/) (2026)

## Abstract (arXiv)

Whilst fact verification has attracted substantial interest in the natural language processing community, verifying misinforming statements against data visualizations such as charts has so far been overlooked. Charts are commonly used in the real-world to summarize and communicate key information, but they can also be easily misused to spread misinformation and promote certain agendas. In this paper, we introduce ChartCheck, a novel, large-scale dataset for explainable fact-checking against real-world charts, consisting of 1.7k charts and 10.5k human-written claims and explanations. We systematically evaluate ChartCheck using vision-language and chart-to-table models, and propose a baseline to the community. Finally, we study chart reasoning types and visual attributes that pose a challenge to these models