mlsanityData to 14 Oct 2026

ChartLens: Fine-grained Visual Attribution in Charts

arXiv 2505.19360 ↗First posted 25 May 2025cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

The growing capabilities of multimodal large language models (MLLMs) have advanced tasks like chart understanding. However, these models often suffer from hallucinations, where generated text sequences conflict with the provided visual data. To address this, we introduce Post-Hoc Visual Attribution for Charts, which identifies fine-grained chart elements that validate a given chart-associated response. We propose ChartLens, a novel chart attribution algorithm that uses segmentation-based techniques to identify chart objects and employs set-of-marks prompting with MLLMs for fine-grained visual attribution. Additionally, we present ChartVA-Eval, a benchmark with synthetic and real-world charts from diverse domains like finance, policy, and economics, featuring fine-grained attribution annotations. Our evaluations show that ChartLens improves fine-grained attributions by 26-66%.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 52 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

34 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs26 Jun
  2. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning4 Jan
  3. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generation27 Nov
  4. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning24 May
  5. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering19 Dec

Other references · 29

  1. 2023Segment Anythingcited by 8,234
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  6. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  7. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  8. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  9. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  10. 2017A simple neural network module for relational reasoningcited by 346
  11. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  12. 2019PlotQA: Reasoning over Scientific Plotscited by 321
17 more
  1. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  2. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  3. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 268
  4. 2021FeTaQA: Free-form Table Question Answeringcited by 175
  5. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  6. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 160
  7. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 112
  8. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
  9. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 98
  10. 2021SciCap: Generating Captions for Scientific Figurescited by 77
  11. 2023A Survey of Large Language Models Attributioncited by 71
  12. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  13. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  14. 2021AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industrycited by 44
  15. 2024The Ethics of Interaction: Mitigating Security Threats in LLMscited by 33
  16. 2023Retrieving Supporting Evidence for Generative Question Answeringcited by 31
  17. 2019Answering Questions about Data Visualizations using Efficient Bimodal Fusioncited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025A Survey of Data Agents: Emerging Paradigm or Overstated Hype?cited by 30
  2. 2026ChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referringcited by 2
  3. 2026LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RADAR: A Reasoning-Guided Attribution Framework for Explainable Visual Data Analysis
  2. 2026ChartAnno: Benchmarking Multimodal Large Language Models for Chart Annotation Generation
  3. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding
  4. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Charts
  5. 2025InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information
  6. 2025START: Spatial and Textual Learning for Chart Understanding
  7. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning
  8. 2026ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models