mlsanityData to 7 Oct 2026

Same Chart, Different Story: Bias in Vision-Language Chart Interpretation

arXiv 2609.26210 ↗First posted 12 Aug 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Vision-language models (VLMs) are increasingly used to interpret charts and generate natural-language explanations for socially consequential data. However, they may produce different narratives for the same chart when only the referenced social group changes, reinforcing stereotypes and misleading decisions. Despite these risks, no benchmark exists for systematically evaluating bias in chart interpretation across social dimensions. We introduce ChartBias, the first benchmark for auditing bias in VLM-based chart interpretation. ChartBias contains 820 manually curated real-world charts spanning six attributes: race, income, age, religion, immigration status, and gender, yielding 4,319 valid chart, attribute instances and 8,638 paired generations where the chart is fixed and only the group term is swapped. Across 12 proprietary and open-source VLMs, totaling 155,484 model responses, we find three widespread failure modes: narrative shift (same chart, different narratives), group hallucination (assigning a chart to a group without evidence), and preference polarity (favourable trends often linked to one group). We further propose a multi-agent mitigation framework that serves as a strong baseline by separating chart-grounded evidence extraction from group-conditioned generation and using a counterfactual judge to verify that group-driven differences are supported by the chart. The framework substantially reduces narrative shift while preserving chart-grounded reasoning. Our findings show that evaluating chart understanding requires measuring not only accuracy, but also fairness and consistency across social groups. We release ChartBias at https://github.com/vis-nlp/ChartBiasBench.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 36 references that are in the map. First posted on arXiv 12 Aug 2026.

Builds on

23 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examples2 Mar
  2. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals30 May

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  5. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  6. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  7. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  8. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  9. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  10. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  11. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 111
  12. 2021Mitigating Language-Dependent Ethnic Bias in BERTcited by 87
9 more
  1. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 77
  2. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  3. 2023Nationality Bias in Text Generationcited by 50
  4. 2024Disclosure and Mitigation of Gender Bias in LLMscited by 47
  5. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  6. 2024Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMscited by 22
  7. 2022Striking a Balance: Reader Takeaways and Preferences when Integrating Text and Chartscited by 17
  8. 2025FAIR-SIGHT: Fairness Assurance in Image Recognition via Simultaneous Conformal Thresholding and Dynamic Output Repaircited by 3
  9. 2025From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Textcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models
  2. 2026Edu-MMBias: A Three-Tier Multimodal Benchmark for Auditing Social Bias in Vision-Language Models under Educational Contexts
  3. 2024BiasDora: Exploring Hidden Biased Associations in Vision-Language Models
  4. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  5. 2025Unveiling the "Fairness Seesaw": Discovering and Mitigating Gender and Race Bias in Vision-Language Models
  6. 2026To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias
  7. 2026Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos
  8. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model