mlsanityData to 14 Oct 2026

Advancing Speech Summarization in Multi-modal LLMs with Reinforcement Learning

arXiv 2509.19631 ↗First posted 23 Sep 2025eess.AS cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Speech summarization is a critical component of spoken content understanding, particularly in the era of rapidly growing spoken and audiovisual data. Recent advances in multi-modal large language models (MLLMs), leveraging the power of LLMs, enable generating textual summaries directly from speech without intermediate transcriptions, while supporting controllable styles and zero-shot generalization. However, open-source MLLMs continue to lag behind the state-of-the-art text-based LLMs, limiting their practical deployment for speech summarization. In this work, we present a novel multi-stage reinforcement learning training framework to enhance the speech summarization capabilities in MLLMs. Our model delivers substantial improvements over strong baselines, outperforms much larger MLLMs, and significantly narrows the gap with state-of-the-art text-based LLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 22 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

14 of 22 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs3 Mar

Other references · 13

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  8. 2024Qwen2-Audio Technical Reportcited by 560
  9. 2021Speech Summarization using Restricted Self-Attentioncited by 23
  10. 2023Leveraging Large Text Corpora for End-to-End Speech Summarizationcited by 11
  11. 2023Transfer Learning from Pre-trained Language Models Improves End-to-End Speech Summarizationcited by 7
  12. 2024Prompting Large Language Models with Audio for General-Purpose Speech Summarizationcited by 6
1 more
  1. 2024An End-to-End Speech Summarization Using Large Language Modelcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management
  2. 2025Mutual Reinforcement of LLM Dialogue Synthesis and Summarization Capabilities for Few-Shot Dialogue Summarization
  3. 2025Explore the Reinforcement Learning for the LLM based ASR and TTS system
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models
  5. 2025Enhancing Speech Large Language Models through Reinforced Behavior Alignment
  6. 2026ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning
  7. 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
  8. 2025Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization