# LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding

arXiv: https://arxiv.org/abs/2306.14924 · HTML: https://mlsanity.com/paper/2306.14924/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 23 Jun 2023.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning](https://mlsanity.com/paper/2304.06588/) (2023)
- [Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks](https://mlsanity.com/paper/2304.10145/) (2023)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)
- [LMentry: A Language Model Benchmark of Elementary Language Tasks](https://mlsanity.com/paper/2211.02069/) (2022)

## Built on by (most cited first, 30 of 64)

- [A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions](https://mlsanity.com/paper/2412.05563/) (2024)
- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra](https://mlsanity.com/paper/2404.03647/) (2024)
- [Network Formation and Dynamics Among Multi-LLMs](https://mlsanity.com/paper/2402.10659/) (2024)
- [Large Language Models for Manufacturing](https://mlsanity.com/paper/2410.21418/) (2024)
- [Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks](https://mlsanity.com/paper/2401.15170/) (2024)
- [Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysis](https://mlsanity.com/paper/2402.01386/) (2024)
- [HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction](https://mlsanity.com/paper/2410.09112/) (2024)
- [FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference](https://mlsanity.com/paper/2505.13109/) (2025)
- [Benchmarking the Capabilities of Large Language Models in Transportation System Engineering: Accuracy, Consistency, and Reasoning Behaviors](https://mlsanity.com/paper/2408.08302/) (2024)
- [From Prompt Engineering to Prompt Science With Human in the Loop](https://mlsanity.com/paper/2401.04122/) (2024)
- [DeepSeek vs. ChatGPT vs. Claude: A Comparative Study for Scientific Computing and Scientific Machine Learning Tasks](https://mlsanity.com/paper/2502.17764/) (2025)
- [Improved prompting and process for writing user personas with LLMs, using qualitative interviews: Capturing behaviour and personality traits of users](https://mlsanity.com/paper/2310.06391/) (2023)
- [From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews](https://mlsanity.com/paper/2312.01202/) (2023)
- [Interpreting Language Reward Models via Contrastive Explanations](https://mlsanity.com/paper/2411.16502/) (2024)
- [From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysis](https://mlsanity.com/paper/2502.11919/) (2025)
- [LLMAuditor: A Framework for Auditing Large Language Models Using Human-in-the-Loop](https://mlsanity.com/paper/2402.09346/) (2024)
- [QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forums](https://mlsanity.com/paper/2405.05345/) (2024)
- [Reflections on Inductive Thematic Saturation as a potential metric for measuring the validity of an inductive Thematic Analysis with LLMs](https://mlsanity.com/paper/2401.03239/) (2024)
- [The Evolution of LLM Adoption in Industry Data Curation Practices](https://mlsanity.com/paper/2412.16089/) (2024)
- [Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors](https://mlsanity.com/paper/2409.18203/) (2024)
- [Labeling Free-text Data using Language Model Ensembles](https://mlsanity.com/paper/2501.08413/) (2025)
- [Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression](https://mlsanity.com/paper/2406.19528/) (2024)
- [A Framework for Collaborating a Large Language Model Tool in Brainstorming for Triggering Creative Thoughts](https://mlsanity.com/paper/2410.11877/) (2024)
- [From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions](https://mlsanity.com/paper/2504.18691/) (2025)
- [DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones](https://mlsanity.com/paper/2511.07427/) (2025)
- [Bridging Dictionary: AI-Generated Dictionary of Partisan Language Use](https://mlsanity.com/paper/2407.09661/) (2024)
- [Using Instruction-Tuned Large Language Models to Identify Indicators of Vulnerability in Police Incident Narratives](https://mlsanity.com/paper/2412.11878/) (2024)
- [Temperature and Persona Shape LLM Agent Consensus With Minimal Accuracy Gains in Qualitative Coding](https://mlsanity.com/paper/2507.11198/) (2025)
- [Long-Document QA with Chain-of-Structured-Thought and Fine-Tuned SLMs](https://mlsanity.com/paper/2603.29232/) (2026)

## Abstract (arXiv)

Deductive coding is a widely used qualitative research method for determining the prevalence of themes across documents. While useful, deductive coding is often burdensome and time consuming since it requires researchers to read, interpret, and reliably categorize a large body of unstructured text documents. Large language models (LLMs), like ChatGPT, are a class of quickly evolving AI tools that can perform a range of natural language processing and reasoning tasks. In this study, we explore the use of LLMs to reduce the time it takes for deductive coding while retaining the flexibility of a traditional content analysis. We outline the proposed approach, called LLM-assisted content analysis (LACA), along with an in-depth case study using GPT-3.5 for LACA on a publicly available deductive coding data set. Additionally, we conduct an empirical benchmark using LACA on 4 publicly available data sets to assess the broader question of how well GPT-3.5 performs across a range of deductive coding tasks. Overall, we find that GPT-3.5 can often perform deductive coding at levels of agreement comparable to human coders. Additionally, we demonstrate that LACA can help refine prompts for deductive coding, identify codes for which an LLM is randomly guessing, and help assess when to use LLMs vs. human coders for deductive coding. We conclude with several implications for future practice of deductive coding and related research methods.