mlsanityData to 14 Oct 2026

Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Charts

arXiv 2505.17374 ↗First posted 23 May 2025cs.HC cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

The field of Multimodal Large Language Models (MLLMs) has made remarkable progress in visual understanding tasks, presenting a vast opportunity to predict the perceptual and emotional impact of charts. However, it also raises concerns, as many applications of LLMs are based on overgeneralized assumptions from a few examples, lacking sufficient validation of their performance and effectiveness. We introduce Chart-to-Experience, a benchmark dataset comprising 36 charts, evaluated by crowdsourced workers for their impact on seven experiential factors. Using the dataset as ground truth, we evaluated capabilities of state-of-the-art MLLMs on two tasks: direct prediction and pairwise comparison of charts. Our findings imply that MLLMs are not as sensitive as human evaluators when assessing individual charts, but are accurate and reliable in pairwise comparisons.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 55 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

24 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generation27 Nov

Other references · 23

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  3. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  4. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  5. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  6. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  7. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  8. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 160
  9. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 125
  10. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 112
  11. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  12. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 78
11 more
  1. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 53
  2. 2023GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognitioncited by 38
  3. 2023DialogueLLM: Context and Emotion Knowledge-Tuned Large Language Models for Emotion Recognition in Conversationscited by 30
  4. 2024Generating Automatic Feedback on UI Mockups with Large Language Modelscited by 23
  5. 2023Enhanced Chart Understanding in Vision and Language Task via Cross-modal Pre-training on Plot Table Pairscited by 15
  6. 2024StimuVAR: Spatiotemporal Stimuli-aware Video Affective Reasoning with Multimodal Large Language Modelscited by 8
  7. 2023Socratis: Are large multimodal models emotionally aware?cited by 5
  8. 2019Data is Personal: Attitudes and Perceptions of Data Visualization in Rural Pennsylvaniacited by 4
  9. 2022Affective Learning Objectives for Communicative Visualizationscited by 4
  10. 2024Assessing the Aesthetic Evaluation Capabilities of GPT-4 with Vision: Insights from Group and Individual Assessmentscited by 4
  11. 2023Affective Visualization Design: Leveraging the Emotional Impact of Datacited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfacescited by 11
  2. 2026Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysiscited by 8
  3. 2025Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Graphical Perception with Multimodal LLMs
  2. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement
  3. 2026ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models
  4. 2025From Charts to Code: A Hierarchical Benchmark for Multimodal Models
  5. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Charts
  6. 2026ChartAnno: Benchmarking Multimodal Large Language Models for Chart Annotation Generation
  7. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning
  8. 2025Towards Understanding Graphical Perception in Large Multimodal Models