mlsanityData to 14 Oct 2026

Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking

arXiv 2301.11843 ↗First posted 27 Jan 2023cs.CL cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Evidence data for automated fact-checking (AFC) can be in multiple modalities such as text, tables, images, audio, or video. While there is increasing interest in using images for AFC, previous works mostly focus on detecting manipulated or fake images. We propose a novel task, chart-based fact-checking, and introduce ChartBERT as the first model for AFC against chart evidence. ChartBERT leverages textual, structural and visual information of charts to determine the veracity of textual claims. For evaluation, we create ChartFC, a new dataset of 15, 886 charts. We systematically evaluate 75 different vision-language (VL) baselines and show that ChartBERT outperforms VL models, achieving 63.8% accuracy. Our results suggest that the task is complex yet feasible, with many challenges ahead.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 41 of its 63 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

41 of 63 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2016Layer Normalizationcited by 5,669
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  11. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  12. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
29 more
  1. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  2. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 363
  3. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  4. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  5. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  6. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  7. 2020Explainable Automated Fact-Checking for Public Health Claimscited by 211
  8. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 196
  9. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  10. 2021FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured informationcited by 167
  11. 2021Automated Fact-Checking for Assisting Human Fact-Checkerscited by 144
  12. 2022End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Modelscited by 107
  13. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  14. 2021X-FACT: A New Benchmark Dataset for Multilingual Fact Checkingcited by 91
  15. 2020INFOTABS: Inference on Tables as Semi-structured Datacited by 86
  16. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  17. 2022Detecting and Understanding Harmful Memes: A Surveycited by 58
  18. 2019LEAF-QA: Locate, Encode & Attend for Figure Question Answeringcited by 54
  19. 2019Fact-Checking Meets Fauxtography: Verifying Claims About Imagescited by 52
  20. 2020XGPT: Cross-modal Generative Pre-Training for Image Captioningcited by 51
  21. 2021A Survey on Multimodal Disinformation Detectioncited by 47
  22. 2021SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS)cited by 35
  23. 2019Answering Questions about Data Visualizations using Efficient Bimodal Fusioncited by 26
  24. 2022Misinformed by Visualization: What Do We Learn From Misinformative Visualizations?cited by 19
  25. 2022Multi-modal Misinformation Detection: Approaches, Challenges and Opportunitiescited by 15
  26. 2021Caption Enriched Samples for Improving Hateful Memes Detectioncited by 14
  27. 2022DisinfoMeme: A Multimodal Dataset for Detecting Meme Intentionally Spreading Out Disinformationcited by 13
  28. 2021Viral Visualizations: How Coronavirus Skeptics Use Orthodox Data Practices to Promote Unorthodox Science Onlinecited by 7
  29. 2021Mapping the Landscape of COVID-19 Crisis Visualizationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 10; 2025: 8; 2026: 11.

35 papers.

Sort:

By area:

  1. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 108
  2. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  3. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 68
  4. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  5. 2023Multimodal Automated Fact-Checking: A Surveycited by 53
  6. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 53
  7. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 46
  8. 2024DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Expertscited by 43
  9. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  10. 2023QACHECK: A Demonstration System for Question-Guided Multi-Hop Fact-Checkingcited by 26
  11. 2023ChartCheck: Explainable Fact-Checking over Real-World Chart Imagescited by 26
  12. 2024DataNarrative: Automated Data-Driven Storytelling with Visualizations and Textscited by 25
23 more
  1. 2024Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQAcited by 19
  2. 2023DOMINO: A Dual-System for Multi-step Visual Language Reasoningcited by 12
  3. 2025MuSciClaims: Multimodal Scientific Claim Verificationcited by 8
  4. 2024Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directionscited by 6
  5. 2025The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Paperscited by 6
  6. 2024Transformers Utilization in Chart Understanding: A Review of Recent Advances & Future Trendscited by 3
  7. 2025Does visualization help AI understand data?cited by 3
  8. 2025ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Chartscited by 2
  9. 2025From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Textcited by 2
  10. 2025Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Frameworkcited by 2
  11. 2025ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinementcited by 2
  12. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancementcited by 1
  13. 2026DATAREEL: Automated Data-Driven Video Story Generation with Animationscited by 1
  14. 2023GenPlot: Increasing the Scale and Diversity of Chart Derendering Datacited by 0
  15. 2026ChartComplete: A Taxonomy-based Inclusive Chart Datasetcited by 0
  16. 2026MEVER: Multi-Modal and Explainable Claim Verification with Graph-based Evidence Retrievalcited by 0
  17. 2026Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessitycited by 0
  18. 2026When Verification Fails: How Compositionally Infeasible Claims Escape Rejectioncited by 0
  19. 2026Beyond Single Plots: A Benchmark for Question Answering on Multi-Chartscited by 0
  20. 2026Assessing Y-Axis Influence: Bias in Multimodal Language Models on Chart-to-Table Translationcited by 0
  21. 2026Chart Deception in Vision-Language Models: From Vulnerability to Mitigationcited by 0
  22. 2026When Can Text Replace Vision? Structural Bottlenecks in Diagram Reasoningcited by 0
  23. 2026Do you "see" what I "see"? A Multi-panel Visual Question Answer Dataset for Large Language Model Chart Analysiscited by 0