mlsanityData to 7 Oct 2026

Quantifying Societal Bias Amplification in Image Captioning

arXiv 2203.15395 ↗First posted 29 Mar 2022cs.CV cs.MMcited by 37

Not on the map. Why?

Abstract (arXiv)

We study societal bias amplification in image captioning. Image captioning models have been shown to perpetuate gender and racial biases, however, metrics to measure, quantify, and evaluate the societal bias in captions are not yet standardized. We provide a comprehensive study on the strengths and limitations of each metric, and propose LIC, a metric to study captioning bias amplification. We argue that, for image captioning, it is not enough to focus on the correct prediction of the protected attribute, and the whole context should be taken into account. We conduct extensive evaluation on traditional and state-of-the-art image captioning models, and surprisingly find that, by only focusing on the protected attribute prediction, bias mitigation models are unexpectedly amplifying bias.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 29 Mar 2022.

Builds on

35 of 51 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
23 more
  1. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  2. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  3. 2016Self-critical Sequence Training for Image Captioningcited by 544
  4. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  5. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  6. 2016Image Captioning with Semantic Attentioncited by 260
  7. 2019Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigationcited by 252
  8. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  9. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  10. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  11. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  12. 2019Does Object Recognition Work for Everyone?cited by 149
  13. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  14. 2018Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddingscited by 114
  15. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  16. 2019Predictive Inequity in Object Detectioncited by 77
  17. 2019Attenuating Bias in Word Vectorscited by 64
  18. 2021Directional Bias Amplificationcited by 54
  19. 2020Mitigating Gender Bias in Captioning Systemscited by 50
  20. 2020Investigating Gender Bias in BERTcited by 40
  21. 2021One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Visioncited by 33
  22. 2020Mitigating Gender Bias Amplification in Distribution by Posterior Regularizationcited by 27
  23. 2021Feature and Label Embedding Spaces Matter in Addressing Image Classifier Biascited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 13; 2024: 9; 2025: 6; 2026: 3.

37 papers.

Sort:

By area:

  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2023The Bias Amplification Paradox in Text-to-Image Generationcited by 69
  3. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 51
  4. 2023Learning Bottleneck Concepts in Image Classificationcited by 49
  5. 2022Gender and Racial Bias in Visual Question Answering Datasetscited by 39
  6. 2023ToViLaG: Your Visual-Language Generative Model is Also An Evildoercited by 27
  7. 2024Would Deep Generative Models Amplify Bias in Future Models?cited by 24
  8. 2022Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Surveycited by 22
  9. 2022Men Also Do Laundry: Multi-Attribute Bias Amplificationcited by 21
  10. 2023Ethical Considerations for Responsible Data Curationcited by 17
  11. 2023Model-Agnostic Gender Debiased Image Captioningcited by 17
  12. 2022Exploring Discrete Diffusion Models for Image Captioningcited by 16
25 more
  1. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decompositioncited by 16
  2. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
  3. 2024Fairness and Bias Mitigation in Computer Vision: A Surveycited by 12
  4. 2025A Large Scale Analysis of Gender Biases in Text-to-Image Generative Modelscited by 12
  5. 2022Improving Fairness in Image Classification via Sketchingcited by 11
  6. 2024FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplicationcited by 11
  7. 2023Adventures of Trustworthy Vision-Language Models: A Surveycited by 10
  8. 2023VLSlice: Interactive Vision-and-Language Slice Discoverycited by 7
  9. 2023ImageCaptioner2: Image Captioner for Image Captioning Bias Amplification Assessmentcited by 5
  10. 2025Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluationcited by 5
  11. 2023Variation of Gender Biases in Visual Recognition Models Before and After Finetuningcited by 2
  12. 2023Gender Bias in Transformer Models: A comprehensive surveycited by 2
  13. 2025From Global to Local: Social Bias Transfer in CLIPcited by 2
  14. 2022Large-Scale Bidirectional Training for Zero-Shot Image Captioningcited by 1
  15. 2023VLIS: Unimodal Language Models Guide Multimodal Language Generationcited by 1
  16. 2024DPA: A one-stop metric to measure bias amplification in classification datasetscited by 1
  17. 2025A Woman with a Knife or A Knife with a Woman? Measuring Directional Bias Amplification in Image Captionscited by 1
  18. 2026Gender Artifacts from Art History to Text-to-Image Generationcited by 1
  19. 2024Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Taskscited by 0
  20. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0
  21. 2024Evaluating authenticity and quality of image captions via sentiment and semantic analysescited by 0
  22. 2025Group-based Distinctive Image Captioning with Memory Difference Encoding and Attentioncited by 0
  23. 2025Multimodal Political Bias Identification and Neutralizationcited by 0
  24. 2026Counting Worlds Branching Time Semantics for post-hoc Bias Mitigation in generative AIcited by 0
  25. 2026Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Modelscited by 0