mlsanityData to 14 Oct 2026

Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models

arXiv 2505.17496 ↗First posted 23 May 2025cs.CL cs.AI cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

End-to-end training of Spoken Language Models (SLMs) commonly involves adapting pre-trained text-based Large Language Models (LLMs) to the speech modality through multi-stage training on diverse tasks such as ASR, TTS and spoken question answering (SQA). Although this multi-stage continual learning equips LLMs with both speech understanding and generation capabilities, the substantial differences in task and data distributions across stages can lead to catastrophic forgetting, where previously acquired knowledge is lost. This paper investigates catastrophic forgetting and evaluates three mitigation strategies-model merging, discounting the LoRA scaling factor, and experience replay to balance knowledge retention with new learning. Results show that experience replay is the most effective, with further gains achieved by combining it with other methods. These findings provide insights for developing more robust and efficient SLM training pipelines.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 36 of its 40 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

36 of 40 references.

References · 36

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  6. 2018Experience Replay for Continual Learningcited by 1,095
  7. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  10. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
24 more
  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  2. 2024Qwen2-Audio Technical Reportcited by 560
  3. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  5. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  6. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  8. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  9. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  10. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  11. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  12. 2023Mitigating the Alignment Tax of RLHFcited by 183
  13. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  14. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  15. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  16. 2023Joint Audio and Speech Understandingcited by 100
  17. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  18. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  19. 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Taskscited by 71
  20. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
  21. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  22. 2024Building a Taiwanese Mandarin Spoken Language Model: A First Attemptcited by 23
  23. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9
  24. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 78
  2. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  3. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  4. 2026Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Modelscited by 2
  5. 2026How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluationcited by 2
  6. 2026Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLMcited by 2
  7. 2026Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervisioncited by 1
  8. 2026ALARM: Audio-Language Alignment for Reasoning Modelscited by 1
  9. 2025Understanding Textual Capability Degradation in Speech LLMs via Parameter Importance Analysiscited by 0
  10. 2025F5-TTS-RO: Extending F5-TTS to Romanian TTS via Lightweight Input Adaptationcited by 0
  11. 2026Adapting Text LLMs to Speech via Multimodal Depth Up-Scalingcited by 0