mlsanityData to 14 Oct 2026

ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement

arXiv 2509.13282 ↗First posted 16 Sep 2025cs.CL cs.CV cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Charts are a crucial visual medium for communicating and representing information. While Large Vision-Language Models (LVLMs) have made progress on chart question answering (CQA), the task remains challenging, particularly when models attend to irrelevant regions of the chart. In this work, we present ChartGaze, a new eye-tracking dataset that captures human gaze patterns during chart reasoning tasks. Through a systematic comparison of human and model attention, we find that LVLMs often diverge from human gaze, leading to reduced interpretability and accuracy. To address this, we propose a gaze-guided attention refinement that aligns image-text attention with human fixations. Our approach improves both answer accuracy and attention alignment, yielding gains of up to 2.56 percentage points across multiple models. These results demonstrate the promise of incorporating human gaze to enhance both the reasoning quality and interpretability of chart-focused LVLMs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 61 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

33 of 61 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  6. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  7. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  8. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  9. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  10. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  11. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  12. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 150
21 more
  1. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  2. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
  3. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  4. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 112
  5. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 108
  6. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  7. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  8. 2023ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Languagecited by 73
  9. 2022OpenCQA: Open-ended Question Answering with Chartscited by 61
  10. 2019LEAF-QA: Locate, Encode & Attend for Figure Question Answeringcited by 54
  11. 2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53
  12. 2025ConceptAttention: Diffusion Transformers Learn Highly Interpretable Featurescited by 52
  13. 2021Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Contentcited by 39
  14. 2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 35
  15. 2023Prompt Highlighter: Interactive Control for Multi-Modal LLMscited by 29
  16. 2023Voila-A: Aligning Vision-Language Models with User's Gaze Attentioncited by 28
  17. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  18. 2022Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignmentcited by 27
  19. 2024On Pre-training of Multimodal Language Models Customized for Chart Understandingcited by 13
  20. 2017Saliency Revisited: Analysis of Mouse Movements versus Fixationscited by 12
  21. 2019Deep Saliency Models : The Quest For The Loss Functioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025ChartAgent: A Chart Understanding Framework with Tool Integrated Reasoningcited by 4
  2. 2026Attention-Guided Saliency Maps for Interpreting Visualization Literacy in VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
  2. 2025In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding
  3. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning
  4. 2025Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Ugly
  5. 2025Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?
  6. 2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering
  7. 2025ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answering
  8. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts