mlsanityData to 14 Oct 2026

Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs

arXiv 2506.08633 ↗First posted 10 Jun 2025eess.AS cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

20 of 27 references.

References · 20

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  6. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  7. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  8. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
  9. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  10. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  11. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  12. 2023SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agentscited by 71
8 more
  1. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  2. 2021Shades of BLEU, Flavours of Success: The Case of MultiWOZcited by 48
  3. 2024Retrieval Augmented End-to-End Spoken Dialog Modelscited by 20
  4. 2024Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forwardcited by 17
  5. 2023Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognitioncited by 12
  6. 2022Speech Aware Dialog System Technology Challenge (DSTC11)cited by 10
  7. 2024Aligning Pre-trained Models for Spoken Language Translationcited by 2
  8. 2024Schema Augmentation for Zero-Shot Domain Adaptation in Dialogue State Trackingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  2. 2026Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projectioncited by 2
  3. 2025The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approachcited by 1
  4. 2025Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Trackingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models as Zero-shot Dialogue State Tracker through Function Calling
  2. 2024SynthDST: Synthetic Data is All You Need for Few-Shot Dialog State Tracking
  3. 2024Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation
  4. 2024A Two-dimensional Zero-shot Dialogue State Tracking Evaluation Method using GPT-4
  5. 2025Spoken Conversational Agents with Large Language Models
  6. 2024Benchmarks Underestimate the Readiness of Multi-lingual Dialogue Agents
  7. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
  8. 2024OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog