mlsanityData to 7 Oct 2026

Women also Snowboard: Overcoming Bias in Captioning Models

arXiv 1803.09797 ↗First posted 26 Mar 2018cs.CVcited by 236

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Most machine learning methods are known to capture and exploit biases of the training data. While some biases are beneficial for learning, others are harmful. Specifically, image captioning models tend to exaggerate biases present in training data (e.g., if a word is present in 60% of training sentences, it might be predicted in 70% of sentences at test time). This can lead to incorrect captions in domains where unbiased captions are desired, or required, due to over-reliance on the learned prior and image context. In this work we investigate generation of gender-specific caption words (e.g. man, woman) based on the person's appearance or the image context. We introduce a new Equalizer model that ensures equal gender probability when gender evidence is occluded in a scene and confident predictions when gender evidence is present. The resulting model is forced to look at a person rather than use contextual cues to make a gender-specific predictions. The losses that comprise our model, the Appearance Confusion Loss and the Confident Loss, are general, and can be added to any description model in order to mitigate impacts of unwanted bias in a description dataset. Our proposed model has lower error than prior work when describing images with people and mentioning their gender and more closely matches the ground truth ratio of sentences including women to sentences including men. We also show that unlike other approaches, our model is indeed more often looking at people when predicting their gender.

Key facts. Cited by 236 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 50; 2022: 64; 2023: 62; 2024: 33; 2025: 16; 2026: 11.

236 papers.

Sort:

By area:

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  4. 2022Video Diffusion Modelscited by 2,280
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  8. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  9. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  10. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  11. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  12. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
38 more
  1. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  2. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  3. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  4. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  5. 2021Gender Bias in Machine Translationcited by 116
  6. 2021Learning Debiased Representation via Disentangled Feature Augmentationcited by 116
  7. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  8. 2021Causal Attention for Vision-Language Taskscited by 111
  9. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  10. 2021EnD: Entangling and Disentangling deep representations for bias correctioncited by 105
  11. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  12. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  13. 2022Text-Only Training for Image Captioning using Noise-Injected CLIPcited by 91
  14. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  15. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  16. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  17. 2021A Study of Face Obfuscation in ImageNetcited by 83
  18. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  19. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  20. 2022A Systematic Study of Bias Amplificationcited by 65
  21. 2022Theories of "Gender" in NLP Bias Researchcited by 62
  22. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
  23. 2022FairPrune: Achieving Fairness Through Pruning for Dermatological Disease Diagnosiscited by 59
  24. 2023Discovering and Mitigating Visual Biases through Keyword Explanationcited by 58
  25. 2021Human-Adversarial Visual Question Answeringcited by 57
  26. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Imagescited by 55
  27. 2021Directional Bias Amplificationcited by 54
  28. 2022Leveling Down in Computer Vision: Pareto Inefficiencies in Fair Deep Classifierscited by 53
  29. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  30. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
  31. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolutioncited by 52
  32. 2021PASS: An ImageNet replacement for self-supervised pretraining without humanscited by 51
  33. 2022Perturbation Augmentation for Fairer NLPcited by 51
  34. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 51
  35. 2023TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Modelscited by 51
  36. 2021Understanding and Mitigating Annotation Bias in Facial Expression Recognitioncited by 50
  37. 2022Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvementcited by 48
  38. 2022Discover and Mitigate Unknown Biases with Debiasing Alternate Networkscited by 47

50 most cited of 236 All 236 (JSON)