Automated Detection and Classification of Delusion-related Content in Naturalistic Audio Diaries Using Multi-Agent Language Models
On the map: LLM agents and tool use → Medical and scientific domain agents
Abstract (arXiv)
Speech monologues recorded in naturalistic settings provide opportunities to characterize mental illness phenomenology and detect symptom exacerbation. Large language models (LLMs) offer new possibilities for automating this process, as they require annotated data primarily for evaluation rather than training. In this paper, we present a novel automated, multi-agent LLM pipeline for the fine-grained, multi-label extraction of language suggestive of delusional beliefs, associated affective responses, and behavioral responses from transcripts of naturalistic audio diaries collected from people with moderate persecutory ideation. Evaluating an ensemble of three foundation models, we demonstrate that detailed diagnostic prompt instructions successfully reduce false positives for delusional theme classification, but also constrain the interpretation of affective or behavioral responses. Furthermore, comparing multi-agent adjudication frameworks shows that complex conversational debate between agents diminishes accuracy on clinically ambiguous text by inducing premature consensus. Instead, majority voting establishes robust performance (Micro F1 of 0.872 and 0.779 for delusion detection and classification respectively). This work provides a validated and scalable pipeline for the automated detection and characterization of content suggesting delusional beliefs in naturalistic speech.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 12 of its 67 references that are in the map. First posted on arXiv 23 May 2026.
Builds on
12 of 67 references.
References · 12
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
- 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
- 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
- 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
- 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
- 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debatecited by 66
- 2023A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Taskscited by 25
- 2024Cognitive Biases in Large Language Models: A Survey and Mitigation Experimentscited by 20
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbots
- 2025Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agents
- 2026Lost in Delusion: Examining LLM Safety Under User Delusions and Distress
- 2025Delusions of Large Language Models
- 2026LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis
- 2026A Multi-Agent Audit Framework for High-Stakes Reasoning: Evaluation and Interpretability in Clinical Mental Health Screening
- 2025MAGI: Multi-Agent Guided Interview for Psychiatric Assessment
- 2026Towards Efficient and Robust Linguistic Emotion Diagnosis for Mental Health via Multi-Agent Instruction Refinement