mlsanityData to 14 Oct 2026

AnchorSIPS: A Synthetic Dataset and Evaluation Resource for Evidence-Supported Psychosis-Risk Symptom Measurement

arXiv 2608.12329 ↗First posted 1 Jun 2026cs.CL cs.AI cs.HCcited by 0

On the map: Synthetic data and self-improvement for LLMs → Medical synthetic data and agents

Abstract (arXiv)

Progress on AI for psychosis-risk assessment is limited by a data-access bottleneck. Real clinical interviews are difficult to share because of privacy, governance, and consent constraints. We present AnchorSIPS, a synthetic dataset of 10K structured psychosis-risk interviews with transcript-grounded measurement targets. Each interview is modeled on Mini-SIPS, a clinician-administered psychosis-risk interview. It captures history, 24 symptom questions, follow-up evidence for items the patient affirms, decisions about delusion-like symptoms (unusual beliefs), hallucination-like symptoms (unusual perceptions), and disorganized communication, exclusion of clear psychotic-level symptoms ("frank psychosis"), and a final attenuated psychosis syndrome (APS) diagnosis, a high-risk state of milder or early psychotic symptoms. The APS diagnosis is not a standalone label. It depends on earlier endorsements, supporting follow-up details, symptom-class decisions, and the frank-psychosis check. Every intermediate decision is anchored to its supporting transcript turns. AnchorSIPS is generated by a plan-then-realize pipeline. A hidden case sheet specifies the patient's clinical state, a deterministic planner fixes the interview structure, and an LLM realizes only the patient utterances under validation and bounded repair. Fixing labels and structure before generation avoids the inter-turn inconsistencies typical of multi-turn LLM dialogue. Across seven LLM baselines, models recover coarse decisions but fail to extract follow-up details or cite supporting transcript turns, so final-label performance overstates interview competence. AnchorSIPS is intended for research on evidence extraction, transcript-grounded measurement, and uncertainty under partial disclosure.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical synthetic data and agents, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 18 of its 44 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

18 of 44 references.

References · 18

  1. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  2. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  3. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  4. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 118
  5. 2024CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counselingcited by 89
  6. 2024A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Reviewcited by 66
  7. 2024Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theorycited by 60
  8. 2023NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notescited by 48
  9. 2025MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistancecited by 39
  10. 2022D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chatcited by 37
  11. 2024AttributionBench: How Hard is Automatic Attribution Evaluation?cited by 26
  12. 2024MDD-5k: A New Diagnostic Conversation Dataset for Mental Disorders Synthesized via Neuro-Symbolic LLM Agentscited by 16
6 more
  1. 2024Synthetic Patient-Physician Dialogue Generation from Clinical Notes Using LLMcited by 15
  2. 2025Crisp: Cognitive Restructuring of Negative Thoughts through Multi-turn Supportive Dialoguescited by 11
  3. 2024DiagESC: Dialogue Synthesis for Integrating Depression Diagnosis into Emotional Support Conversationcited by 6
  4. 2025CliniChat: A Multi-Source Knowledge-Driven Framework for Clinical Interview Dialogue Reconstruction and Evaluationcited by 4
  5. 2026CHiRPE: A Step Towards Real-World Clinical NLP with Clinician-Oriented Model Explanationscited by 2
  6. 2025From Medical Records to Diagnostic Dialogues: A Clinical-Grounded Approach and Dataset for Psychiatric Comorbiditycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming
  2. 2026Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis
  3. 2026DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbots
  4. 2026Ask Before You Diagnose: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry
  5. 2026Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safety
  6. 2025Diagnosing Hallucination Risk in AI Surgical Decision-Support: A Sequential Framework for Sequential Validation
  7. 2025LLAMADRS: Evaluating Open-Source LLMs on Real Clinical Interviews--To Reason or Not to Reason?
  8. 2024MedHallBench: A New Benchmark for Assessing Hallucination in Medical Large Language Models