# From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Text

arXiv: https://arxiv.org/abs/2508.09450 · HTML: https://mlsanity.com/paper/2508.09450/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 68 references that are in the map.
- First posted on arXiv 13 Aug 2025.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://mlsanity.com/paper/2305.15005/) (2023)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- [DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models](https://mlsanity.com/paper/2202.04053/) (2022)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Towards Debiasing Sentence Representations](https://mlsanity.com/paper/2007.08100/) (2020)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings](https://mlsanity.com/paper/1904.04047/) (2019)
- [Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges](https://mlsanity.com/paper/2311.03287/) (2023)
- [Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications](https://mlsanity.com/paper/2108.02818/) (2021)
- [Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement](https://mlsanity.com/paper/2402.11436/) (2024)
- [Towards Controllable Biases in Language Generation](https://mlsanity.com/paper/2005.00268/) (2020)
- [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021)
- [Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model](https://mlsanity.com/paper/2010.09142/) (2020)
- [OpenCQA: Open-ended Question Answering with Charts](https://mlsanity.com/paper/2210.06628/) (2022)
- [Understanding Undesirable Word Embedding Associations](https://mlsanity.com/paper/1908.06361/) (2019)

## Built on by (most cited first, 2 of 2)

- [Chart Deception in Vision-Language Models: From Vulnerability to Mitigation](https://mlsanity.com/paper/2607.22600/) (2026)
- [Same Chart, Different Story: Bias in Vision-Language Chart Interpretation](https://mlsanity.com/paper/2609.26210/) (2026)

## Abstract (arXiv)

Charts are very common for exploring data and communicating insights, but extracting key takeaways from charts and articulating them in natural language can be challenging. The chart-to-text task aims to automate this process by generating textual summaries of charts. While with the rapid advancement of large Vision-Language Models (VLMs), we have witnessed great progress in this domain, little to no attention has been given to potential biases in their outputs. This paper investigates how VLMs can amplify geo-economic biases when generating chart summaries, potentially causing societal harm. Specifically, we conduct a large-scale evaluation of geo-economic biases in VLM-generated chart summaries across 6,000 chart-country pairs from six widely used proprietary and open-source models to understand how a country's economic status influences the sentiment of generated summaries. Our analysis reveals that existing VLMs tend to produce more positive descriptions for high-income countries compared to middle- or low-income countries, even when country attribution is the only variable changed. We also find that models such as GPT-4o-mini, Gemini-1.5-Flash, and Phi-3.5 exhibit varying degrees of bias. We further explore inference-time prompt-based debiasing techniques using positive distractors but find them only partially effective, underscoring the complexity of the issue and the need for more robust debiasing strategies. Our code and dataset are publicly available here.