# When do Word Embeddings Accurately Reflect Surveys on our Beliefs About People?

arXiv: https://arxiv.org/abs/2004.12043 · HTML: https://mlsanity.com/paper/2004.12043/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2020.

## Built on by (most cited first, 11 of 11)

- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [A Survey of Race, Racism, and Anti-Racism in NLP](https://mlsanity.com/paper/2106.11410/) (2021)
- [Knowledge of cultural moral norms in large language models](https://mlsanity.com/paper/2306.01857/) (2023)
- [American == White in Multimodal Language-and-Image AI](https://mlsanity.com/paper/2207.00691/) (2022)
- [Theoretical foundations and limits of word embeddings: what types of meaning can they capture?](https://mlsanity.com/paper/2107.10413/) (2021)
- [No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media](https://mlsanity.com/paper/2211.03634/) (2022)
- [Speciesist Language and Nonhuman Animal Bias in English Masked Language Models](https://mlsanity.com/paper/2203.05140/) (2022)
- [Measuring Dimensions of Self-Presentation in Twitter Bios and their Links to Misinformation Sharing](https://mlsanity.com/paper/2305.09548/) (2023)
- [Speciesism in Natural Language Processing Research](https://mlsanity.com/paper/2410.14194/) (2024)
- [Does Local News Stay Local?: Online Content Shifts in Sinclair-Acquired Stations](https://mlsanity.com/paper/2510.07060/) (2025)
- [How Do We Engage with Other Disciplines? A Framework to Study Meaningful Interdisciplinary Discourse in Scholarly Publications](https://mlsanity.com/paper/2601.17020/) (2026)

## Abstract (arXiv)

Social biases are encoded in word embeddings. This presents a unique opportunity to study society historically and at scale, and a unique danger when embeddings are used in downstream applications. Here, we investigate the extent to which publicly-available word embeddings accurately reflect beliefs about certain kinds of people as measured via traditional survey methods. We find that biases found in word embeddings do, on average, closely mirror survey data across seventeen dimensions of social meaning. However, we also find that biases in embeddings are much more reflective of survey data for some dimensions of meaning (e.g. gender) than others (e.g. race), and that we can be highly confident that embedding-based measures reflect survey data only for the most salient biases.