# Position: Thematic Analysis of Unstructured Clinical Transcripts with Large Language Models

arXiv: https://arxiv.org/abs/2509.14597 · HTML: https://mlsanity.com/paper/2509.14597/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 42 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs](https://mlsanity.com/paper/2401.11314/) (2024)
- [How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment](https://mlsanity.com/paper/2309.14049/) (2023)
- [Cost-of-Pass: An Economic Framework for Evaluating Language Models](https://mlsanity.com/paper/2504.13359/) (2025)
- [Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks](https://mlsanity.com/paper/2401.15170/) (2024)
- [Potential and Perils of Large Language Models as Judges of Unstructured Textual Data](https://mlsanity.com/paper/2501.08167/) (2025)
- [Auto-TA: Towards Scalable Automated Thematic Analysis (TA) via Multi-Agent Large Language Models with Reinforcement Learning](https://mlsanity.com/paper/2506.23998/) (2025)
- [BEYONDWORDS is All You Need: Agentic Generative AI based Social Media Themes Extractor](https://mlsanity.com/paper/2503.01880/) (2025)
- [Automated Thematic Analyses Using LLMs: Xylazine Wound Management Social Media Chatter Use Case](https://mlsanity.com/paper/2507.10803/) (2025)

## Built on by (most cited first, 2 of 2)

- [SFT-TA: Supervised Fine-Tuned Agents in Multi-Agent LLMs for Automated Inductive Thematic Analysis](https://mlsanity.com/paper/2509.17167/) (2025)
- [Automated Thematic Analysis for Clinical Qualitative Data: Iterative Codebook Refinement with Full Provenance](https://mlsanity.com/paper/2603.08989/) (2026)

## Abstract (arXiv)

This position paper examines how large language models (LLMs) can support thematic analysis of unstructured clinical transcripts, a widely used but resource-intensive method for uncovering patterns in patient and provider narratives. We conducted a systematic review of recent studies applying LLMs to thematic analysis, complemented by an interview with a practicing clinician. Our findings reveal that current approaches remain fragmented across multiple dimensions including types of thematic analysis, datasets, prompting strategies and models used, most notably in evaluation. Existing evaluation methods vary widely (from qualitative expert review to automatic similarity metrics), hindering progress and preventing meaningful benchmarking across studies. We argue that establishing standardized evaluation practices is critical for advancing the field. To this end, we propose an evaluation framework centered on three dimensions: validity, reliability, and interpretability.