# TS-OPD: Reconciling ASR and QA in Speech Language Models via Task-Specific On-Policy Distillation

arXiv: https://arxiv.org/abs/2609.29464 · HTML: https://mlsanity.com/paper/2609.29464/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Audio language models and spoken dialogue, one of 11 directions in Speech and audio language models.
- Builds on 22 of its 24 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Placed in

- Speech and audio language models / [Audio language models and spoken dialogue](https://mlsanity.com/speech/audio-language-models-and-spoken-dialogue/)

## Builds on (most cited first, 22 of 22)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation](https://mlsanity.com/paper/2407.05361/) (2024)
- [Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM](https://mlsanity.com/paper/2411.00774/) (2024)
- [Qwen3-ASR Technical Report](https://mlsanity.com/paper/2601.21337/) (2026)
- [DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data](https://mlsanity.com/paper/2409.20007/) (2024)
- [ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark](https://mlsanity.com/paper/2507.05727/) (2025)
- [AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM](https://mlsanity.com/paper/2412.01145/) (2024)
- [Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation](https://mlsanity.com/paper/2605.16826/) (2026)
- [Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition](https://mlsanity.com/paper/2210.15282/) (2022)
- [SSR: Alignment-Aware Modality Connector for Speech Language Models](https://mlsanity.com/paper/2410.00168/) (2024)
- [X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs](https://mlsanity.com/paper/2603.24596/) (2026)
- [Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models](https://mlsanity.com/paper/2505.17496/) (2025)
- [BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillation](https://mlsanity.com/paper/2405.19041/) (2024)
- [Cross-Modal Knowledge Distillation for Speech Large Language Models](https://mlsanity.com/paper/2509.14930/) (2025)
- [StepAudio 2.5 Technical Report](https://mlsanity.com/paper/2605.23463/) (2026)
- [X3-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment](https://mlsanity.com/paper/2607.21550/) (2026)
- [Language-Specialized Multi-Teacher On-Policy Distillation for Multilingual LLM-Based ASR](https://mlsanity.com/paper/2608.03610/) (2026)
- [On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin](https://mlsanity.com/paper/2608.11898/) (2026)
- [SALMONN-2: Advancing General-Purpose Hearing Abilities with Self-Supervised Representations](https://mlsanity.com/paper/2607.17079/) (2026)

## Abstract (arXiv)

Speech Language Models (SLMs) inherit strong instruction-following capabilities from pretrained language models, yet ASR specialization can substantially degrade them. To address this ASR--QA trade-off, we propose Task-Specific On-Policy Distillation (TS-OPD), which leverages models before and after ASR specialization as complementary QA and ASR teachers. The student generates separate task-conditioned trajectories for ASR and QA, each supervised only by its corresponding teacher, thereby reducing direct competition between the two supervision signals. Experiments on basic ASR, contextual ASR, and QA demonstrate that TS-OPD improves recognition while preserving QA capability. Moreover, TS-OPD remains robust across different balancing coefficients and continues to benefit from increased distillation data.