# Reflections on Inductive Thematic Saturation as a potential metric for measuring the validity of an inductive Thematic Analysis with LLMs

arXiv: https://arxiv.org/abs/2401.03239 · HTML: https://mlsanity.com/paper/2401.03239/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 47 references that are in the map.
- First posted on arXiv 6 Jan 2024.

## Builds on (most cited first, 12 of 12)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)
- [LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding](https://mlsanity.com/paper/2306.14924/) (2023)
- [CoAIcoder: Examining the Effectiveness of AI-assisted Human-to-Human Collaboration in Qualitative Analysis](https://mlsanity.com/paper/2304.05560/) (2023)
- [Using Large Language Models to Support Thematic Analysis in Empirical Legal Studies](https://mlsanity.com/paper/2310.18729/) (2023)
- [Shaping the Emerging Norms of Using Large Language Models in Social Computing Research](https://mlsanity.com/paper/2307.04280/) (2023)
- [Writing user personas with Large Language Models: Testing phase 6 of a Thematic Analysis of semi-structured interviews](https://mlsanity.com/paper/2305.18099/) (2023)
- [Theory In, Theory Out: The uses of social theory in machine learning for social science](https://mlsanity.com/paper/2001.03203/) (2020)
- [Qualitative Analysis for Human Centered AI](https://mlsanity.com/paper/2112.03784/) (2021)
- [Algorithmic failure as a humanities methodology: machine learning's mispredictions identify rich cases for qualitative analysis](https://mlsanity.com/paper/2305.11663/) (2023)
- [Critical-Reflective Human-AI Collaboration: Exploring Computational Tools for Art Historical Image Retrieval](https://mlsanity.com/paper/2306.12843/) (2023)
- [Theme and Topic: How Qualitative Research and Topic Modeling Can Be Brought Together](https://mlsanity.com/paper/2210.00707/) (2022)

## Built on by (most cited first, 6 of 6)

- [A Computational Method for Measuring "Open Codes" in Qualitative Analysis](https://mlsanity.com/paper/2411.12142/) (2024)
- [Codebook Reduction and Saturation: Novel observations on Inductive Thematic Saturation for Large Language Models and initial coding in Thematic Analysis](https://mlsanity.com/paper/2503.04859/) (2025)
- [Can machines perform a qualitative data analysis? Reading the debate with Alan Turing](https://mlsanity.com/paper/2512.04121/) (2025)
- [Multi-LLM Thematic Analysis with Dual Reliability Metrics: Combining Cohen's Kappa and Semantic Similarity for Qualitative Research Validation](https://mlsanity.com/paper/2512.20352/) (2025)
- [Whose Story Gets Told? Positionality and Bias in LLM Summaries of Life Narratives](https://mlsanity.com/paper/2604.20131/) (2026)
- [How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding](https://mlsanity.com/paper/2609.11109/) (2026)

## Abstract (arXiv)

This paper presents a set of reflections on saturation and the use of Large Language Models (LLMs) for performing Thematic Analysis (TA). The paper suggests that initial thematic saturation (ITS) could be used as a metric to assess part of the transactional validity of TA with LLM, focusing on the initial coding. The paper presents the initial coding of two datasets of different sizes, and it reflects on how the LLM reaches some form of analytical saturation during the coding. The procedure proposed in this work leads to the creation of two codebooks, one comprising the total cumulative initial codes and the other the total unique codes. The paper proposes a metric to synthetically measure ITS using a simple mathematical calculation employing the ratio between slopes of cumulative codes and unique codes. The paper contributes to the initial body of work exploring how to perform qualitative analysis with LLMs.