mlsanityData to 14 Oct 2026

How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding

arXiv 2609.11109 ↗First posted 10 Sep 2026cs.HC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The utility of AI in multi-coder qualitative coding has been widely discussed, yet little empirical evidence exists to delineate the contexts in which it performs reliably. We address this gap by quantifying the effectiveness of multi-agent LLM coding across varied qualitative datasets, revealing key contextual and structural factors that mediate coding outcomes. We developed a literature-informed baseline pipeline that enables AI agents to independently code, debate, and reconcile disagreements. Results revealed that coding accuracy depends on factors such as codebook length, qualitative data similarity, and agent disagreement. Notably, intense and unresolved debates between agents led to higher accuracy. Our analysis showed that while LLMs emulate many human discussion behaviors, they lack adaptive responsiveness to context. From these findings, we offer design recommendations for building automated coding systems. Our open-source AI discussion dataset and methodological framework lay the groundwork for advancing the design of AI-mediated automated thematic analysis.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 162 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

39 of 162 references.

References · 39

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  3. 2019Learning Loss for Active Learningcited by 336
  4. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  5. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 222
  6. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 159
  7. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  8. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 114
  9. 2023Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Codingcited by 82
  10. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  11. 2023LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysiscited by 70
  12. 2023LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Codingcited by 64
27 more
  1. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 47
  2. 2022Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approachescited by 39
  3. 2023CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Modelscited by 34
  4. 2017Anyone Can Become a Troll: Causes of Trolling Behavior in Online Discussionscited by 30
  5. 2024Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concernscited by 28
  6. 2023CoAIcoder: Examining the Effectiveness of AI-assisted Human-to-Human Collaboration in Qualitative Analysiscited by 25
  7. 2023Using Large Language Models for Qualitative Analysis can Introduce Serious Biascited by 22
  8. 2024Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Taskscited by 22
  9. 2023Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysiscited by 16
  10. 2023Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the modelcited by 15
  11. 2024Automating Thematic Analysis: How LLMs Analyse Controversial Topicscited by 10
  12. 2023Text Alignment Is An Efficient Unified Model for Massive NLP Taskscited by 8
  13. 2023QualiGPT: GPT as an easy-to-use tool for qualitative codingcited by 8
  14. 2023Harnessing ChatGPT for thematic analysis: Are we ready?cited by 8
  15. 2024QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forumscited by 8
  16. 2024Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineeringcited by 8
  17. 2020Discourse Parsing of Contentious, Non-Convergent Online Discussionscited by 6
  18. 2022"Mama Always Had a Way of Explaining Things So I Could Understand'': A Dialogue Corpus for Learning to Construct Explanationscited by 6
  19. 2024Reflections on Inductive Thematic Saturation as a potential metric for measuring the validity of an inductive Thematic Analysis with LLMscited by 6
  20. 2024Thematic Analysis with Open-Source Generative AI and Machine Learning: A New Method for Inductive Qualitative Codebook Developmentcited by 6
  21. 2024A Computational Method for Measuring "Open Codes" in Qualitative Analysiscited by 6
  22. 2024Decoding Complexity: Exploring Human-AI Concordance in Qualitative Codingcited by 4
  23. 2024Comprehensive Evaluation of Large Language Models for Topic Modelingcited by 3
  24. 2025Optimizing Generative AI's Accuracy and Transparency in Inductive Thematic Analysis: A Human-AI Comparisoncited by 3
  25. 2024A Framework For Discussing LLMs as Tools for Qualitative Analysiscited by 2
  26. 2024Generative AI Tools in Academic Research: Applications and Implications for Qualitative and Quantitative Research Methodologiescited by 2
  27. 2023Impact of Human-AI Interaction on User Trust and Reliance in AI-Assisted Qualitative Codingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Temperature and Persona Shape LLM Agent Consensus With Minimal Accuracy Gains in Qualitative Coding
  2. 2025Enhancing LLM Performance Through Debate: An Empirical Study on Multi-Agent Debate for Coding Tasks
  3. 2026Disagreement as Data: Reasoning Trace Analytics in Multi-Agent Systems
  4. 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
  5. 2025A Survey of Vibe Coding with Large Language Models
  6. 2026From Tool to Teammate: LLM Coding Agents as Collaborative Partners for Behavioral Labeling in Educational Dialogue Analysis
  7. 2026A Phased Workflow for Operating LLM-Based Coding Agents
  8. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding