mlsanityData to 7 Oct 2026

VisText: A Benchmark for Semantically Rich Chart Captioning

arXiv 2307.05356 ↗First posted 28 Jun 2023cs.CV cs.HC cs.LGcited by 111

Not on the map. Why?

Cited by 58 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Captions that describe or explain charts help improve recall and comprehension of the depicted data and provide a more accessible medium for people with visual disabilities. However, current approaches for automatically generating such captions struggle to articulate the perceptual or cognitive features that are the hallmark of charts (e.g., complex trends and patterns). In response, we introduce VisText: a dataset of 12,441 pairs of charts and captions that describe the charts' construction, report key statistics, and identify perceptual and cognitive phenomena. In VisText, a chart is available as three representations: a rasterized image, a backing data table, and a scene graph -- a hierarchical representation of a chart's visual elements akin to a web page's Document Object Model (DOM). To evaluate the impact of VisText, we fine-tune state-of-the-art language models on our chart captioning task and apply prefix-tuning to produce captions that vary the semantic content they convey. Our models generate coherent, semantically rich captions and perform on par with state-of-the-art chart captioning models across machine translation and text generation metrics. Through qualitative analysis, we identify six broad categories of errors that our models make that can inform future work.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Builds on 29 of its 79 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

29 of 79 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  10. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  11. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  12. 2017Challenges in Data-to-Document Generationcited by 226
17 more
  1. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  2. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  3. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 77
  4. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  5. 2021Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Contentcited by 38
  6. 2019Multimodal Machine Translation through Visuals and Speechcited by 29
  7. 2019Figure Captioning with Reasoning and Sequence-Level Trainingcited by 21
  8. 2018Chart-Text: A Fully Automated Chart Image Descriptorcited by 20
  9. 2022LineCap: Line Charts for Data Visualization Captioning Modelscited by 17
  10. 2022Striking a Balance: Reader Takeaways and Preferences when Integrating Text and Chartscited by 17
  11. 2021Computer Vision and Conflicting Values: Describing People with Automated Alt Textcited by 14
  12. 2018DataSite: Proactive Visual Data Exploration with Computation of Insight-based Recommendationscited by 12
  13. 2021Towards Understanding How Readers Integrate Charts and Captions: A Case Study with Line Chartscited by 12
  14. 2021Kori: Interactive Synthesis of Text and Charts in Data Documentscited by 10
  15. 2018Ethical Dimensions of Visualization Researchcited by 9
  16. 2021Communicating Visualizations without Visuals: Investigation of Visualization Alternative Text for People with Visual Impairmentscited by 4
  17. 2022How Well Do My Results Generalize Now? The External Validity of Online Privacy and Security Surveyscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 30; 2025: 44; 2026: 29.

111 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  4. 2024What matters when building vision-language models?cited by 321
  5. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  6. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  7. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  8. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  9. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  10. 2025PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Modelcited by 84
  11. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  12. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
38 more
  1. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  2. 2025FineVision: Open Data Is All You Needcited by 51
  3. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
  4. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 47
  5. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  6. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 35
  7. 2023mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Modelcited by 31
  8. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  9. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  10. 2024DataNarrative: Automated Data-Driven Storytelling with Visualizations and Textscited by 25
  11. 2023GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captionscited by 21
  12. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  13. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  14. 2023SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructionscited by 19
  15. 2024Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQAcited by 18
  16. 2025NVIDIA Nemotron Nano V2 VLcited by 15
  17. 2025InterChat: Enhancing Generative Visual Analytics using Multimodal Interactionscited by 13
  18. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
  19. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Chartscited by 11
  20. 2024DOGR: Towards Versatile Visual Document Grounding and Referringcited by 10
  21. 2025Boosting Chart-to-Code Generation in MLLM via Dual Preference-Guided Refinementcited by 10
  22. 2025ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Datasetcited by 9
  23. 2025MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuningcited by 9
  24. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  25. 2025MuSciClaims: Multimodal Scientific Claim Verificationcited by 8
  26. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  27. 2025PatentLMM: Large Multimodal Model for Generating Descriptions for Patent Figurescited by 7
  28. 2025See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMscited by 7
  29. 2025ChartCap: Mitigating Hallucination of Dense Chart Captioningcited by 7
  30. 2024SciCapenter: Supporting Caption Composition for Scientific Figures with Machine-Generated Captions and Ratingscited by 5
  31. 2024Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directionscited by 5
  32. 2025ChartCards: A Chart-Metadata Generation Framework for Multi-Task Chart Understandingcited by 5
  33. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMscited by 5
  34. 2025Same or Not? Enhancing Visual Perception in Vision-Language Modelscited by 5
  35. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understandingcited by 5
  36. 2024ChartAdapter: Large Vision-Language Model for Chart Summarizationcited by 4
  37. 2025Towards Understanding Graphical Perception in Large Multimodal Modelscited by 4
  38. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularizationcited by 4

50 most cited of 111 All 111 (JSON)