# Meaning Is Not A Metric: Using LLMs to make cultural context legible at scale

arXiv: https://arxiv.org/abs/2505.23785 · HTML: https://mlsanity.com/paper/2505.23785/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 121 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 31)

- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [The science of fake news](https://mlsanity.com/paper/2307.07903/) (2023)
- [Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties](https://mlsanity.com/paper/2309.00779/) (2023)
- [An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels](https://mlsanity.com/paper/2203.11364/) (2022)
- [Aligning to Thousands of Preferences via System Message Generalization](https://mlsanity.com/paper/2405.17977/) (2024)
- [Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs](https://mlsanity.com/paper/2503.08688/) (2025)
- [MindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling](https://mlsanity.com/paper/2310.05231/) (2023)
- [Diverging Preferences: When do Annotators Disagree and do Models Know?](https://mlsanity.com/paper/2410.14632/) (2024)
- [Culture is Not Trivia: Sociocultural Theory for Cultural NLP](https://mlsanity.com/paper/2502.12057/) (2025)
- [Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models](https://mlsanity.com/paper/2306.04746/) (2023)
- [Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.16534/) (2025)
- [Engagement, User Satisfaction, and the Amplification of Divisive Content on Social Media](https://mlsanity.com/paper/2305.16941/) (2023)
- [Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations](https://mlsanity.com/paper/2504.03274/) (2025)
- [From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models](https://mlsanity.com/paper/2305.17174/) (2023)
- [Embedding Democratic Values into Social Media AIs via Societal Objective Functions](https://mlsanity.com/paper/2307.13912/) (2023)
- [MindScape Study: Integrating LLM and Behavioral Sensing for Personalized AI-Driven Journaling Experiences](https://mlsanity.com/paper/2409.09570/) (2024)
- [Evaluating Generative AI Systems is a Social Science Measurement Challenge](https://mlsanity.com/paper/2411.10939/) (2024)
- [Building Human Values into Recommender Systems: An Interdisciplinary Synthesis](https://mlsanity.com/paper/2207.10192/) (2022)
- [The Case for "Thick Evaluations" of Cultural Representation in AI](https://mlsanity.com/paper/2503.19075/) (2025)
- [Can Language Models Represent the Past without Anachronism?](https://mlsanity.com/paper/2505.00030/) (2025)
- [A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions](https://mlsanity.com/paper/2502.11095/) (2025)
- [Provocations from the Humanities for Generative AI Research](https://mlsanity.com/paper/2502.19190/) (2025)
- [A Matter of Perspective(s): Contrasting Human and LLM Argumentation in Subjective Decision-Making on Subtle Sexism](https://mlsanity.com/paper/2502.14052/) (2025)
- [Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness](https://mlsanity.com/paper/2502.09637/) (2025)
- [Writing user personas with Large Language Models: Testing phase 6 of a Thematic Analysis of semi-structured interviews](https://mlsanity.com/paper/2305.18099/) (2023)
- [Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles](https://mlsanity.com/paper/2406.06840/) (2024)
- [Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks](https://mlsanity.com/paper/2408.01346/) (2024)
- [Decoding Complexity: Exploring Human-AI Concordance in Qualitative Coding](https://mlsanity.com/paper/2403.06607/) (2024)
- [Deconstructing Depression Stigma: Integrating AI-driven Data Collection and Analysis with Causal Knowledge Graphs](https://mlsanity.com/paper/2502.06075/) (2025)

## Built on by (most cited first, 10 of 10)

- [Multi-Actor Generative Artificial Intelligence as a Game Engine](https://mlsanity.com/paper/2507.08892/) (2025)
- [Computational Hermeneutics: Evaluating generative AI as a cultural technology](https://mlsanity.com/paper/2604.16403/) (2026)
- [Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value](https://mlsanity.com/paper/2512.03399/) (2025)
- [Why Slop Matters](https://mlsanity.com/paper/2601.06060/) (2025)
- [AI as Entertainment](https://mlsanity.com/paper/2601.08768/) (2026)
- [Semiotic Complexity and Its Epistemological Implications for Modeling Culture](https://mlsanity.com/paper/2508.00095/) (2025)
- [The Differential Meaning of Models: A Framework for Analyzing the Structural Consequences of Semantic Modeling Decisions](https://mlsanity.com/paper/2509.00248/) (2025)
- [Language Models as Measurement Apparatus for Culture](https://mlsanity.com/paper/2607.02459/) (2026)
- [The House with a Million Windows: Interactive Fiction for Narrative Restorying](https://mlsanity.com/paper/2609.12537/) (2026)
- [Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picture](https://mlsanity.com/paper/2609.12575/) (2026)

## Abstract (arXiv)

This position paper argues that large language models (LLMs) can make cultural context, and therefore human meaning, legible at an unprecedented scale in AI-based sociotechnical systems. We argue that such systems have previously been unable to represent human meaning because they rely on thin descriptions (numerical representations that enforce standardization and therefore strip human activity of the cultural context which gives it meaning). By contrast, scholars in the humanities and qualitative social sciences have developed frameworks for representing meaning through thick description (verbal representations that accommodate heterogeneity and retain contextual information needed to represent human meaning). The verbal capabilities of LLMs now provide a means of at least partially automating the generation and processing of thick descriptions, offering new ways to deploy them at scale. We argue that the problem of rendering human meaning legible is not just about selecting better metrics but about developing new representational formats based on thick description. We frame this as a crucial direction for the application of generative AI and identify five key challenges: preserving context, maintaining interpretive pluralism, integrating perspectives based on lived experience and critical distance, distinguishing qualitative content from quantitative magnitude, and acknowledging meaning as dynamic rather than static.