# Prompts versus Rules: Auditing and Controlling Speech Naturalness Behaviors in Voice User Simulators

arXiv: https://arxiv.org/abs/2610.11015 · HTML: https://mlsanity.com/paper/2610.11015/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 18 of 18)

- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [Generative Spoken Dialogue Language Modeling](https://mlsanity.com/paper/2203.16502/) (2022)
- [Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction](https://mlsanity.com/paper/2502.11946/) (2025)
- [PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models](https://mlsanity.com/paper/2602.06053/) (2026)
- [Following Length Constraints in Instructions](https://mlsanity.com/paper/2406.17744/) (2024)
- [τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains](https://mlsanity.com/paper/2603.13686/) (2026)
- [Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency](https://mlsanity.com/paper/2604.04847/) (2026)
- [Towards human-like spoken dialogue generation between AI agents from written dialogue](https://mlsanity.com/paper/2310.01088/) (2023)
- [EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents](https://mlsanity.com/paper/2605.13841/) (2026)
- [VAmoS Bench: Voice Agent Simulation Bench](https://mlsanity.com/paper/2607.27453/) (2026)
- [IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows](https://mlsanity.com/paper/2606.19595/) (2026)
- [EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions](https://mlsanity.com/paper/2604.16456/) (2026)
- [F-Actor: Controllable Conversational Behaviour in Full-Duplex Models](https://mlsanity.com/paper/2601.11329/) (2026)
- [TriageSim: A Conversational Emergency Triage Simulation Framework from Structured Electronic Health Records](https://mlsanity.com/paper/2603.10035/) (2026)
- [SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue](https://mlsanity.com/paper/2603.16783/) (2026)

## Abstract (arXiv)

As voice agents gain more popularity commercially, the user simulators used to evaluate the deployed agents are also being developed to include more realistic, variable, and diverse speech naturalness behaviors -- disfluency, interruption and backchanneling. The quality of the user simulator directly affects the validity of agent evaluation results. However, we find that most studies so far have not examined in detail whether the intended configuration for these behaviors is realized in the simulation. In this study, we audit the realized naturalness behaviors of tau-Voice, our own LLM-based prompting approach across three models, and our rule-based injection algorithm for disfluency, interruption, and backchanneling. We find that prompting for these behaviors is unreliable and produces speech inconsistent with the instructions, placed and distributed less naturally than the instruction implies. In contrast, our rule-based, model-free algorithm produces controllable and diverse naturalness behaviors more aligned with natural speech. Our results suggest that LLMs not purpose-trained for user simulation are not sufficient on their own to represent authentic user behavior, and that linguistically informed deterministic approaches or specialized models are needed to close the gap; auditing and reporting realized naturalness behaviors, rather than configured settings, is what makes that gap visible.