# INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detection

arXiv: https://arxiv.org/abs/2602.18448 · HTML: https://mlsanity.com/paper/2602.18448/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 92 references that are in the map.
- First posted on arXiv 28 Jan 2026.

## Builds on (most cited first, 19 of 19)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs](https://mlsanity.com/paper/2501.17399/) (2025)
- [Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach](https://mlsanity.com/paper/2407.16833/) (2024)
- [LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization](https://mlsanity.com/paper/2301.13298/) (2023)
- [DialFact: A Benchmark for Fact-Checking in Dialogue](https://mlsanity.com/paper/2110.08222/) (2021)
- [DynaEval: Unifying Turn and Dialogue Level Evaluation](https://mlsanity.com/paper/2106.01112/) (2021)
- [STAR: A Schema-Guided Dialog Dataset for Transfer Learning](https://mlsanity.com/paper/2010.11853/) (2020)
- [GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theory](https://mlsanity.com/paper/2406.11149/) (2024)
- [SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation](https://mlsanity.com/paper/2406.12975/) (2024)
- [ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models](https://mlsanity.com/paper/2503.00564/) (2025)
- [LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigm](https://mlsanity.com/paper/2502.19103/) (2025)
- [FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation](https://mlsanity.com/paper/2210.13832/) (2022)
- [Multi-lingual Multi-turn Automated Red Teaming for LLMs](https://mlsanity.com/paper/2504.03174/) (2025)
- [Automated Survey Collection with LLM-based Conversational Agents](https://mlsanity.com/paper/2504.02891/) (2025)
- [Real-World En Call Center Transcripts Dataset with PII Redaction](https://mlsanity.com/paper/2507.02958/) (2025)
- [All Required, In Order: Phase-Level Evaluation for AI-Human Dialogue in Healthcare and Beyond](https://mlsanity.com/paper/2601.08690/) (2026)

## Built on by (most cited first, 1 of 1)

- [Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error](https://mlsanity.com/paper/2609.14400/) (2026)

## Abstract (arXiv)

Administrative phone tasks drain roughly 1 trillion USD annually from U.S. healthcare, with over 500 million insurance-benefit verification calls manually handled in 2024. We introduce INSURE-Dial, to our knowledge the first public benchmark for developing and assessing compliance-aware voice agents for phase-aware call auditing with span-based compliance verification. The corpus includes 50 de-identified, AI-initiated calls with live insurance representatives (mean 71 turns/call) and 1,000 synthetically generated calls that mirror the same workflow. All calls are annotated with a phase-structured JSON schema covering IVR navigation, patient identification, coverage status, medication checks (up to two drugs), and agent identification (CRN), and each phase is labeled for Information and Procedural compliance under explicit ask/answer logic. We define two novel evaluation tasks: (1) Phase Boundary Detection (span segmentation under phase-specific acceptance rules) and (2) Compliance Verification (IC/PC decisions given fixed spans). Per-phase scores are strong across small, low-latency baselines, but end-to-end reliability is constrained by span-boundary errors. On real calls, full-call exact segmentation is low, showing a gap between conversational fluency and audit-grade evidence.