# AnchorSIPS: A Synthetic Dataset and Evaluation Resource for Evidence-Supported Psychosis-Risk Symptom Measurement

arXiv: https://arxiv.org/abs/2608.12329 · HTML: https://mlsanity.com/paper/2608.12329/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical synthetic data and agents, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Medical synthetic data and agents](https://mlsanity.com/synth/medical-synthetic-data-and-agents/)

## Builds on (most cited first, 18 of 18)

- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation](https://mlsanity.com/paper/2306.02022/) (2023)
- [CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling](https://mlsanity.com/paper/2405.16433/) (2024)
- [A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review](https://mlsanity.com/paper/2405.02559/) (2024)
- [Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory](https://mlsanity.com/paper/2407.03103/) (2024)
- [NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notes](https://mlsanity.com/paper/2310.15959/) (2023)
- [MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance](https://mlsanity.com/paper/2503.13509/) (2025)
- [D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chat](https://mlsanity.com/paper/2205.11764/) (2022)
- [AttributionBench: How Hard is Automatic Attribution Evaluation?](https://mlsanity.com/paper/2402.15089/) (2024)
- [MDD-5k: A New Diagnostic Conversation Dataset for Mental Disorders Synthesized via Neuro-Symbolic LLM Agents](https://mlsanity.com/paper/2408.12142/) (2024)
- [Synthetic Patient-Physician Dialogue Generation from Clinical Notes Using LLM](https://mlsanity.com/paper/2408.06285/) (2024)
- [Crisp: Cognitive Restructuring of Negative Thoughts through Multi-turn Supportive Dialogues](https://mlsanity.com/paper/2504.17238/) (2025)
- [DiagESC: Dialogue Synthesis for Integrating Depression Diagnosis into Emotional Support Conversation](https://mlsanity.com/paper/2408.06044/) (2024)
- [CliniChat: A Multi-Source Knowledge-Driven Framework for Clinical Interview Dialogue Reconstruction and Evaluation](https://mlsanity.com/paper/2504.10418/) (2025)
- [CHiRPE: A Step Towards Real-World Clinical NLP with Clinician-Oriented Model Explanations](https://mlsanity.com/paper/2601.18102/) (2026)
- [From Medical Records to Diagnostic Dialogues: A Clinical-Grounded Approach and Dataset for Psychiatric Comorbidity](https://mlsanity.com/paper/2510.25232/) (2025)

## Abstract (arXiv)

Progress on AI for psychosis-risk assessment is limited by a data-access bottleneck. Real clinical interviews are difficult to share because of privacy, governance, and consent constraints. We present AnchorSIPS, a synthetic dataset of 10K structured psychosis-risk interviews with transcript-grounded measurement targets. Each interview is modeled on Mini-SIPS, a clinician-administered psychosis-risk interview. It captures history, 24 symptom questions, follow-up evidence for items the patient affirms, decisions about delusion-like symptoms (unusual beliefs), hallucination-like symptoms (unusual perceptions), and disorganized communication, exclusion of clear psychotic-level symptoms ("frank psychosis"), and a final attenuated psychosis syndrome (APS) diagnosis, a high-risk state of milder or early psychotic symptoms. The APS diagnosis is not a standalone label. It depends on earlier endorsements, supporting follow-up details, symptom-class decisions, and the frank-psychosis check. Every intermediate decision is anchored to its supporting transcript turns. AnchorSIPS is generated by a plan-then-realize pipeline. A hidden case sheet specifies the patient's clinical state, a deterministic planner fixes the interview structure, and an LLM realizes only the patient utterances under validation and bounded repair. Fixing labels and structure before generation avoids the inter-turn inconsistencies typical of multi-turn LLM dialogue. Across seven LLM baselines, models recover coarse decisions but fail to extract follow-up details or cite supporting transcript turns, so final-label performance overstates interview competence. AnchorSIPS is intended for research on evidence extraction, transcript-grounded measurement, and uncertainty under partial disclosure.