# Language Models as Measurement Apparatus for Culture

arXiv: https://arxiv.org/abs/2607.02459 · HTML: https://mlsanity.com/paper/2607.02459/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 69 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [The Geometry of Culture: Analyzing Meaning through Word Embeddings](https://mlsanity.com/paper/1803.09288/) (2018)
- [Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge](https://mlsanity.com/paper/2502.00561/) (2025)
- [Culture is Not Trivia: Sociocultural Theory for Cultural NLP](https://mlsanity.com/paper/2502.12057/) (2025)
- [Discovering Differences in the Representation of People using Contextualized Semantic Axes](https://mlsanity.com/paper/2210.12170/) (2022)
- [Meaning Is Not A Metric: Using LLMs to make cultural context legible at scale](https://mlsanity.com/paper/2505.23785/) (2025)
- [On Classification with Large Language Models in Cultural Analytics](https://mlsanity.com/paper/2410.12029/) (2024)
- [Findings on Conversation Disentanglement](https://mlsanity.com/paper/2112.05346/) (2021)
- [Subversive Characters and Stereotyping Readers: Characterizing Queer Relationalities with Dialogue-Based Relation Extraction](https://mlsanity.com/paper/2410.14978/) (2024)
- [What is a protest anyway? Codebook conceptualization is still a first-order concern in LLM-era classification](https://mlsanity.com/paper/2510.03541/) (2025)

## Abstract (arXiv)

Language models are increasingly used to quantify cultural phenomena, but what makes such measurement distinctively cultural? This paper argues that NLP work on culture is a material-discursive practice: the apparatus -- model, data, annotation, evaluation -- participates in constituting the cultural reality it measures, rather than passively recording it. Drawing on Karen Barad's concept of the agential cut -- the contingent boundary between phenomenon and instrument -- I show that the apparatus's substantive design choices draw such boundaries, and that the boundary is entangled from the start because language models have already internalized much of the cultural material they measure. I illustrate this through three case studies on television and film dialogue (measuring structure, interaction, and deviation) and three examinations of the apparatus itself (erasure of cultural markers, attunement to historical material, and agency in an agentic workflow). This big picture analysis proposes a research program that is theory-driven, empirically rigorous, and culturally contingent, treating each agential cut as a conscious commitment, at once methodological and ethical.