# Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour

arXiv: https://arxiv.org/abs/2209.12786 · HTML: https://mlsanity.com/paper/2209.12786/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 26 Sep 2022.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Meaning without reference in large language models](https://mlsanity.com/paper/2208.02957/) (2022)
- [Visual Commonsense in Pretrained Unimodal and Multimodal Models](https://mlsanity.com/paper/2205.01850/) (2022)
- [The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color](https://mlsanity.com/paper/2110.08182/) (2021)

## Built on by (most cited first, 5 of 5)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Cabbage Sweeter than Cake? Analysing the Potential of Large Language Models for Learning Conceptual Spaces](https://mlsanity.com/paper/2310.05481/) (2023)
- [Ranking Entities along Conceptual Space Dimensions with LLMs: An Analysis of Fine-Tuning Strategies](https://mlsanity.com/paper/2402.15337/) (2024)
- [Color Bind: Exploring Color Perception in Text-to-Image Models](https://mlsanity.com/paper/2508.19791/) (2025)
- [Extracting Conceptual Spaces from LLMs Using Prototype Embeddings](https://mlsanity.com/paper/2509.19269/) (2025)

## Abstract (arXiv)

Language models (LMs) trained on raw texts have no direct access to the physical world. Gordon and Van Durme (2013) point out that LMs can thus suffer from reporting bias: texts rarely report on common facts, instead focusing on the unusual aspects of a situation. If LMs are only trained on text corpora and naively memorise local co-occurrence statistics, they thus naturally would learn a biased view of the physical world. While prior studies have repeatedly verified that LMs of smaller scales (e.g., RoBERTa, GPT-2) amplify reporting bias, it remains unknown whether such trends continue when models are scaled up. We investigate reporting bias from the perspective of colour in larger language models (LLMs) such as PaLM and GPT-3. Specifically, we query LLMs for the typical colour of objects, which is one simple type of perceptually grounded physical common sense. Surprisingly, we find that LLMs significantly outperform smaller LMs in determining an object's typical colour and more closely track human judgments, instead of overfitting to surface patterns stored in texts. This suggests that very large models of language alone are able to overcome certain types of reporting bias that are characterized by local co-occurrences.