# Automating Thematic Analysis: How LLMs Analyse Controversial Topics

arXiv: https://arxiv.org/abs/2405.06919 · HTML: https://mlsanity.com/paper/2405.06919/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 96 references that are in the map.
- First posted on arXiv 11 May 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Verbosity Bias in Preference Labeling by Large Language Models](https://mlsanity.com/paper/2310.10076/) (2023)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)
- [LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis](https://mlsanity.com/paper/2310.15100/) (2023)
- [CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models](https://mlsanity.com/paper/2304.07366/) (2023)
- [CoAIcoder: Examining the Effectiveness of AI-assisted Human-to-Human Collaboration in Qualitative Analysis](https://mlsanity.com/paper/2304.05560/) (2023)
- [Intersectional Bias in Causal Language Models](https://mlsanity.com/paper/2107.07691/) (2021)
- [Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysis](https://mlsanity.com/paper/2309.10771/) (2023)
- [Truth Machines: Synthesizing Veracity in AI Language Models](https://mlsanity.com/paper/2301.12066/) (2023)
- [Generative AI: Implications and Applications for Education](https://mlsanity.com/paper/2305.07605/) (2023)
- [A Sign That Spells: DALL-E 2, Invisual Images and The Racial Politics of Feature Space](https://mlsanity.com/paper/2211.06323/) (2022)
- [Towards social generative AI for education: theory, practices and ethics](https://mlsanity.com/paper/2306.10063/) (2023)

## Built on by (most cited first, 10 of 10)

- [TAMA: A Human-AI Collaborative Thematic Analysis Framework Using Multi-Agent LLMs for Clinical Interviews](https://mlsanity.com/paper/2503.20666/) (2025)
- [Auto-TA: Towards Scalable Automated Thematic Analysis (TA) via Multi-Agent Large Language Models with Reinforcement Learning](https://mlsanity.com/paper/2506.23998/) (2025)
- [LOGOS: LLM-driven End-to-End Grounded Theory Development and Schema Induction for Qualitative Research](https://mlsanity.com/paper/2509.24294/) (2025)
- [Investigating Expert-in-the-Loop LLM Discourse Patterns for Ancient Intertextual Analysis](https://mlsanity.com/paper/2409.01882/) (2024)
- [Temperature and Persona Shape LLM Agent Consensus With Minimal Accuracy Gains in Qualitative Coding](https://mlsanity.com/paper/2507.11198/) (2025)
- [Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Risks](https://mlsanity.com/paper/2603.05801/) (2026)
- [Processes Matter: How ML/GAI Approaches Could Support Open Qualitative Coding of Online Discourse Datasets](https://mlsanity.com/paper/2504.02887/) (2025)
- [How Trustworthy Are LLM-as-Judge Ratings for Interpretive Responses? Implications for Qualitative Research Workflows](https://mlsanity.com/paper/2604.00008/) (2026)
- [Bridging Human Interpretation and Machine Representation: A Landscape of Qualitative Data Analysis in the LLM Era](https://mlsanity.com/paper/2601.11739/) (2026)
- [How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding](https://mlsanity.com/paper/2609.11109/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are promising analytical tools. They can augment human epistemic, cognitive and reasoning abilities, and support 'sensemaking', making sense of a complex environment or subject by analysing large volumes of data with a sensitivity to context and nuance absent in earlier text processing systems. This paper presents a pilot experiment that explores how LLMs can support thematic analysis of controversial topics. We compare how human researchers and two LLMs GPT-4 and Llama 2 categorise excerpts from media coverage of the controversial Australian Robodebt scandal. Our findings highlight intriguing overlaps and variances in thematic categorisation between human and machine agents, and suggest where LLMs can be effective in supporting forms of discourse and thematic analysis. We argue LLMs should be used to augment, and not replace human interpretation, and we add further methodological insights and reflections to existing research on the application of automation to qualitative research methods. We also introduce a novel card-based design toolkit, for both researchers and practitioners to further interrogate LLMs as analytical tools.