# ML-EAT: A Multilevel Embedding Association Test for Interpretable and Transparent Social Science

arXiv: https://arxiv.org/abs/2408.01966 · HTML: https://mlsanity.com/paper/2408.01966/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 69 references that are in the map.
- First posted on arXiv 4 Aug 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://mlsanity.com/paper/2012.15859/) (2020)
- [Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings](https://mlsanity.com/paper/1904.04047/) (2019)
- [Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases](https://mlsanity.com/paper/2010.15052/) (2020)
- [Measuring Social Biases in Grounded Vision and Language Embeddings](https://mlsanity.com/paper/2002.08911/) (2020)
- [Understanding Undesirable Word Embedding Associations](https://mlsanity.com/paper/1908.06361/) (2019)
- [Contrastive Language-Vision AI Models Pretrained on Web-Scraped Multimodal Data Exhibit Sexual Objectification Bias](https://mlsanity.com/paper/2212.11261/) (2022)
- [American == White in Multimodal Language-and-Image AI](https://mlsanity.com/paper/2207.00691/) (2022)
- [Evidence for Hypodescent in Visual Semantic AI](https://mlsanity.com/paper/2205.10764/) (2022)
- [Global Voices, Local Biases: Socio-Cultural Prejudices across Languages](https://mlsanity.com/paper/2310.17586/) (2023)
- [Evaluating Biased Attitude Associations of Language Models in an Intersectional Context](https://mlsanity.com/paper/2307.03360/) (2023)
- [VAST: The Valence-Assessing Semantics Test for Contextualizing Language Models](https://mlsanity.com/paper/2203.07504/) (2022)
- [Large scale analysis of gender bias and sexism in song lyrics](https://mlsanity.com/paper/2208.02052/) (2022)
- [Disentangling Societal Inequality from Model Biases: Gender Inequality in Divorce Court Proceedings](https://mlsanity.com/paper/2307.10200/) (2023)

## Built on by (most cited first, 3 of 3)

- [Fragments to Facts: Partial-Information Fragment Inference from LLMs](https://mlsanity.com/paper/2505.13819/) (2025)
- [RPAM: A Principled Metric for Evaluating Associations in Language Models with High Predictive Validity in Downstream Outputs](https://mlsanity.com/paper/2607.05679/) (2026)
- [Calibrating WEAT Against Anisotropy: ZCA Whitening as a Geometric Pre-Processing Step for Embedding Association Tests](https://mlsanity.com/paper/2608.06908/) (2026)

## Abstract (arXiv)

This research introduces the Multilevel Embedding Association Test (ML-EAT), a method designed for interpretable and transparent measurement of intrinsic bias in language technologies. The ML-EAT addresses issues of ambiguity and difficulty in interpreting the traditional EAT measurement by quantifying bias at three levels of increasing granularity: the differential association between two target concepts with two attribute concepts; the individual effect size of each target concept with two attribute concepts; and the association between each individual target concept and each individual attribute concept. Using the ML-EAT, this research defines a taxonomy of EAT patterns describing the nine possible outcomes of an embedding association test, each of which is associated with a unique EAT-Map, a novel four-quadrant visualization for interpreting the ML-EAT. Empirical analysis of static and diachronic word embeddings, GPT-2 language models, and a CLIP language-and-image model shows that EAT patterns add otherwise unobservable information about the component biases that make up an EAT; reveal the effects of prompting in zero-shot models; and can also identify situations when cosine similarity is an ineffective metric, rendering an EAT unreliable. Our work contributes a method for rendering bias more observable and interpretable, improving the transparency of computational investigations into human minds and societies.