mlsanityData to 14 Oct 2026

CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents

arXiv 2606.21453 ↗First posted 19 Jun 2026cs.HC cs.AI cs.SDcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Task-oriented voice agents need to map spoken user requests to structured outputs such as semantic frames, executable actions, and function calls. A common approach is to cascade ASR with a text-based LLM, but transcription errors can propagate to downstream structured output generation, especially under noisy conditions. Spoken language models (SLMs) offer a direct speech-based alternative, yet adapting them to new tasks typically requires paired speech-target annotations. Motivated by this gap, we present CORTIS, a text-only adaptation framework for task-oriented voice agents. CORTIS fine-tunes SLMs using text-form task supervision, enabling speech-based structured output generation at inference time without task-specific speech-target annotations during adaptation. We evaluate CORTIS on two Qwen2.5-Omni backbones and three task-oriented speech datasets, including an in-house product dataset, and compare it with matched ASR-LLM cascades trained with the same text-form task supervision. Results show that CORTIS performs competitively with matched cascades and offers clearer advantages under acoustic degradation, particularly in preserving high-level task semantics. These findings suggest that text-only fine-tuning of SLMs can serve as a practical adaptation strategy for voice agents when paired speech-target data are costly to collect.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 33 of its 39 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

33 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use17 Apr
  2. 2025AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks29 Jun

Other references · 31

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  6. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  8. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  9. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  10. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  11. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  12. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
19 more
  1. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  2. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  3. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 71
  4. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  5. 2021A Survey on Spoken Language Understanding: Recent Advances and New Frontierscited by 53
  6. 2024Octopus v2: On-device language model for super agentcited by 47
  7. 2020Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systemscited by 38
  8. 2024BLSP-Emo: Towards Empathetic Large Speech-Language Modelscited by 36
  9. 2024Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speechcited by 18
  10. 2021Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systemscited by 15
  11. 2024Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networkscited by 8
  12. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  13. 2025Measuring the Effect of Transcription Noise on Downstream Language Understanding Taskscited by 7
  14. 2025Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuningcited by 7
  15. 2025Exploring Fine-Tuning of Large Audio Language Models for Spoken Language Understanding under Limited Speech Datacited by 6
  16. 2024Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuningcited by 4
  17. 2023Multimodal Audio-textual Architecture for Robust Spoken Language Understandingcited by 3
  18. 2025DESAMO: A Device for Elder-Friendly Smart Homes Powered by Embedded LLM with Audio Modalitycited by 3
  19. 2025SpeechLLMs for Large-scale Contextualized Zero-shot Slot Fillingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Spoken Conversational Agents with Large Language Models
  2. 2026Talk2Agent: Benchmarking Voice Interfaces for Text Agents
  3. 2026The Evolving Bottleneck in Speech Generation: Interface Co-design and Staged Alignment from CosyVoice to Qwen-Audio-3.0-TTS ↗
  4. 2026SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning
  5. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
  6. 2026AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach
  7. 2026AuRA: Internalizing Audio Understanding into LLMs as LoRA
  8. 2025VoiceAgentBench: Are Voice Assistants ready for agentic tasks?