When Does a Spoken Agent Have Enough Evidence to Act? The PACT-SLM Contract Test
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks also fits: Social simulation and multi-agent debate
Abstract (arXiv)
Streaming spoken agents may produce the correct final action after acting too early. Final-turn scores do not reveal whether each observed speech prefix supports an exposed action. We introduce the Partial Speech Action Contract for Turn Taking in Speech Language Models (PACT-SLM), a controlled test that assigns the first valid action time and evaluates both action identity and timing. In the primary test, 80 paired contrast groups from four held-out semantic families yield 1,600 prefix predictions across clean and 15 dB noise renderings. Using source-utterance semantic targets rather than counterbalanced branch codes, WavLM Base Plus reaches 26.03% pooled post-onset semantic-label accuracy (95% group-bootstrap interval [22.14%, 29.68%]), exposes an action on 18.99% of pre-onset prefixes, and predicts 5.94% of complete trajectories exactly. Its pooled label score is at the 96th percentile of 100 within-prefix label permutations, below the 97.5th-percentile reference (26.73%). It exceeds matched text, scalar-acoustic, and shuffled-representation probes in pooled post-onset label accuracy. Elapsed time is more onset-exact (36.25% versus 23.13%) but less accurate about action identity (9.92% versus 26.03%). These results motivate separate measurement of action identity and onset timing in partial-speech evaluations.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 12 of its 14 references that are in the map. First posted on arXiv 28 Sep 2026.
Builds on
12 of 14 references.
References · 12
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
- 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 107
- 2025Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examinercited by 38
- 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 30
- 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
- 2025SHANKS: Simultaneous Hearing and Thinking for Spoken Language Modelscited by 16
- 2025FLEXI: Benchmarking Full-duplex Human-LLM Speech Interactioncited by 14
- 2026EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion Reasoningcited by 13
- 2024AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLMcited by 11
- 2025Benchmarking Contextual and Paralinguistic Reasoning in Speech-LLMs: A Case Study with In-the-Wild Datacited by 10
- 2025ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Modelscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents
- 2026Talk2Agent: Benchmarking Voice Interfaces for Text Agents
- 2025Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents
- 2026Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception
- 2026When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories
- 2026VoiceLongMemEval: Do Assistants Remember How You Sounded?
- 2026Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes
- 2025AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding