# On Classification with Large Language Models in Cultural Analytics

arXiv: https://arxiv.org/abs/2410.12029 · HTML: https://mlsanity.com/paper/2410.12029/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 66 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [Calibrated Language Models Must Hallucinate](https://mlsanity.com/paper/2311.14648/) (2023)
- [Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense](https://mlsanity.com/paper/2405.04655/) (2024)
- [Racial categories in machine learning](https://mlsanity.com/paper/1811.11668/) (2018)
- [Confabulation: The Surprising Value of Large Language Model Hallucinations](https://mlsanity.com/paper/2406.04175/) (2024)
- [Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings](https://mlsanity.com/paper/2309.08591/) (2023)
- [AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters](https://mlsanity.com/paper/2401.06408/) (2024)
- [Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets](https://mlsanity.com/paper/2406.18906/) (2024)
- [Living Machines: A study of atypical animacy](https://mlsanity.com/paper/2005.11140/) (2020)

## Built on by (most cited first, 9 of 9)

- [What is a protest anyway? Codebook conceptualization is still a first-order concern in LLM-era classification](https://mlsanity.com/paper/2510.03541/) (2025)
- [AI Fiction in the Wild](https://mlsanity.com/paper/2606.22748/) (2026)
- [A Comparative Analysis of Word Segmentation, Part-of-Speech Tagging, and Named Entity Recognition for Historical Chinese Sources, 1900-1950](https://mlsanity.com/paper/2503.19844/) (2025)
- [Semiotic Complexity and Its Epistemological Implications for Modeling Culture](https://mlsanity.com/paper/2508.00095/) (2025)
- [FicSim: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fiction](https://mlsanity.com/paper/2510.20926/) (2025)
- [Estranged Predictions: Measuring Semantic Category Disruption with Masked Language Modelling](https://mlsanity.com/paper/2511.08109/) (2025)
- [ARGUS: Seeing the Influence of Narrative Features on Persuasion in Argumentative Texts](https://mlsanity.com/paper/2602.24109/) (2026)
- ['The Order in the Horse's Heart': A Case Study in LLM-Assisted Stylometry for the Discovery of Biblical Allusion in Modern Literary Fiction](https://mlsanity.com/paper/2604.19447/) (2026)
- [Language Models as Measurement Apparatus for Culture](https://mlsanity.com/paper/2607.02459/) (2026)

## Abstract (arXiv)

In this work, we survey the way in which classification is used as a sensemaking practice in cultural analytics, and assess where large language models can fit into this landscape. We identify ten tasks supported by publicly available datasets on which we empirically assess the performance of LLMs compared to traditional supervised methods, and explore the ways in which LLMs can be employed for sensemaking goals beyond mere accuracy. We find that prompt-based LLMs are competitive with traditional supervised models for established tasks, but perform less well on de novo tasks. In addition, LLMs can assist sensemaking by acting as an intermediary input to formal theory testing.