# Text Annotation via Inductive Coding: Comparing Human Experts to LLMs in Qualitative Data Analysis

arXiv: https://arxiv.org/abs/2512.00046 · HTML: https://mlsanity.com/paper/2512.00046/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 48 references that are in the map.
- First posted on arXiv 17 Nov 2025.

## Builds on (most cited first, 9 of 9)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)
- [Close to Human-Level Agreement: Tracing Journeys of Violent Speech in Incel Posts with GPT-4-Enhanced Annotations](https://mlsanity.com/paper/2401.02001/) (2024)
- [Automating the Information Extraction from Semi-Structured Interview Transcripts](https://mlsanity.com/paper/2403.04819/) (2024)

## Built on by (most cited first, 4 of 4)

- [How to Interpret Agent Behavior](https://mlsanity.com/paper/2605.13625/) (2026)
- [greCAPTCHA: Assessing Understanding as Evidence of Research Authorship Under Generative AI](https://mlsanity.com/paper/2609.20481/) (2026)
- [Bridging Human Interpretation and Machine Representation: A Landscape of Qualitative Data Analysis in the LLM Era](https://mlsanity.com/paper/2601.11739/) (2026)
- [From Variance to Invariance: Qualitative Content Analysis for Narrative Graph Annotation](https://mlsanity.com/paper/2603.01930/) (2026)

## Abstract (arXiv)

This paper investigates the automation of qualitative data analysis, focusing on inductive coding using large language models (LLMs). Unlike traditional approaches that rely on deductive methods with predefined labels, this research investigates the inductive process where labels emerge from the data. The study evaluates the performance of six open-source LLMs compared to human experts. As part of the evaluation, experts rated the perceived difficulty of the quotes they coded. The results reveal a peculiar dichotomy: human coders consistently perform well when labeling complex sentences but struggle with simpler ones, while LLMs exhibit the opposite trend. Additionally, the study explores systematic deviations in both human and LLM generated labels by comparing them to the golden standard from the test set. While human annotations may sometimes differ from the golden standard, they are often rated more favorably by other humans. In contrast, some LLMs demonstrate closer alignment with the true labels but receive lower evaluations from experts.