# The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color

arXiv: https://arxiv.org/abs/2110.08182 · HTML: https://mlsanity.com/paper/2110.08182/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 15 Oct 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision](https://mlsanity.com/paper/2010.06775/) (2020)
- [Inducing Relational Knowledge from BERT](https://mlsanity.com/paper/1911.12753/) (2019)
- [Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels](https://mlsanity.com/paper/1512.06974/) (2015)
- [PROST: Physical Reasoning of Objects through Space and Time](https://mlsanity.com/paper/2106.03634/) (2021)
- [Stereotyping and Bias in the Flickr30K Dataset](https://mlsanity.com/paper/1605.06083/) (2016)
- [Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations](https://mlsanity.com/paper/1910.01157/) (2019)
- [Evaluating representations by the complexity of learning low-loss predictors](https://mlsanity.com/paper/2009.07368/) (2020)

## Built on by (most cited first, 30 of 32)

- [Linearly Mapping from Image to Text Space](https://mlsanity.com/paper/2209.15162/) (2022)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Mind's Eye: Grounded Language Model Reasoning through Simulation](https://mlsanity.com/paper/2210.05359/) (2022)
- [MIB: A Mechanistic Interpretability Benchmark](https://mlsanity.com/paper/2504.13151/) (2025)
- [Event knowledge in large language models: the gap between the impossible and the unlikely](https://mlsanity.com/paper/2212.01488/) (2022)
- [Visual Commonsense in Pretrained Unimodal and Multimodal Models](https://mlsanity.com/paper/2205.01850/) (2022)
- [DesCo: Learning Object Recognition with Rich Language Descriptions](https://mlsanity.com/paper/2306.14060/) (2023)
- [Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding](https://mlsanity.com/paper/2303.12513/) (2023)
- [Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization](https://mlsanity.com/paper/2206.01532/) (2022)
- [Defining Knowledge: Bridging Epistemology and Large Language Models](https://mlsanity.com/paper/2410.02499/) (2024)
- [Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour](https://mlsanity.com/paper/2209.12786/) (2022)
- [Visually-augmented pretrained language models for NLP tasks without images](https://mlsanity.com/paper/2212.07937/) (2022)
- [Substance or Style: What Does Your Image Embedding Know?](https://mlsanity.com/paper/2307.05610/) (2023)
- [The Zero Body Problem: Probing LLM Use of Sensory Language](https://mlsanity.com/paper/2504.06393/) (2025)
- [VIPHY: Probing "Visible" Physical Commonsense Knowledge](https://mlsanity.com/paper/2209.07000/) (2022)
- [How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input?](https://mlsanity.com/paper/2209.08982/) (2022)
- [Understanding Natural Language Understanding Systems. A Critical Analysis](https://mlsanity.com/paper/2303.04229/) (2023)
- [What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge](https://mlsanity.com/paper/2205.07065/) (2022)
- [Cabbage Sweeter than Cake? Analysing the Potential of Large Language Models for Learning Conceptual Spaces](https://mlsanity.com/paper/2310.05481/) (2023)
- [Ranking Entities along Conceptual Space Dimensions with LLMs: An Analysis of Fine-Tuning Strategies](https://mlsanity.com/paper/2402.15337/) (2024)
- [Diagnosing Vision Language Models' Perception by Leveraging Human Methods for Color Vision Deficiencies](https://mlsanity.com/paper/2505.17461/) (2025)
- [Color Bind: Exploring Color Perception in Text-to-Image Models](https://mlsanity.com/paper/2508.19791/) (2025)
- [Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction](https://mlsanity.com/paper/2210.12905/) (2022)
- [Visually Grounded Commonsense Knowledge Acquisition](https://mlsanity.com/paper/2211.12054/) (2022)
- [Implicit Affordance Acquisition via Causal Action-Effect Modeling in the Video Domain](https://mlsanity.com/paper/2312.11345/) (2023)
- [WinoViz: Probing Visual Properties of Objects Under Different States](https://mlsanity.com/paper/2402.13584/) (2024)
- [Estimating Commonsense Plausibility through Semantic Shifts](https://mlsanity.com/paper/2502.13464/) (2025)
- [Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobs](https://mlsanity.com/paper/2505.02072/) (2025)
- [Everyday Physics in Korean Contexts: A Culturally Grounded Physical Reasoning Benchmark](https://mlsanity.com/paper/2509.17807/) (2025)
- [Position: Natural Language Should Not Fully Replace Formal Languages](https://mlsanity.com/paper/2607.20432/) (2026)

## Abstract (arXiv)

Recent work has raised concerns about the inherent limitations of text-only pretraining. In this paper, we first demonstrate that reporting bias, the tendency of people to not state the obvious, is one of the causes of this limitation, and then investigate to what extent multimodal training can mitigate this issue. To accomplish this, we 1) generate the Color Dataset (CoDa), a dataset of human-perceived color distributions for 521 common objects; 2) use CoDa to analyze and compare the color distribution found in text, the distribution captured by language models, and a human's perception of color; and 3) investigate the performance differences between text-only and multimodal models on CoDa. Our results show that the distribution of colors that a language model recovers correlates more strongly with the inaccurate distribution found in text than with the ground-truth, supporting the claim that reporting bias negatively impacts and inherently limits text-only training. We then demonstrate that multimodal models can leverage their visual training to mitigate these effects, providing a promising avenue for future research.