mlsanityData to 7 Oct 2026

An End-to-End Speech Summarization Using Large Language Model

arXiv 2407.02005 ↗First posted 2 Jul 2024cs.CL cs.SD eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Abstractive Speech Summarization (SSum) aims to generate human-like text summaries from spoken content. It encounters difficulties in handling long speech input and capturing the intricate cross-modal mapping between long speech inputs and short text summaries. Research on large language models (LLMs) and multimodal information fusion has provided new insights for addressing these challenges. In this paper, we propose an end-to-end SSum model that utilizes Q-Former as a connector for the audio-text modality and employs LLMs to generate text summaries directly from speech features. We adopt a multi-stage training approach that includes LLM based ASR and Text Summarization (TSum) tasks as auxiliary tasks. ASR tasks are used to align feature spaces and enhance the LLM's ability to handle longer speech. Then, we utilize a curriculum learning strategy to facilitate the model's transition from TSum to SSum. Finally, our model achieves competitive performance on the How-2 dataset.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

19 of 30 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  10. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  11. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  12. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
7 more
  1. 2019Multimodal Abstractive Summarization for How2 Videoscited by 41
  2. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 34
  3. 2021Speech Summarization using Restricted Self-Attentioncited by 23
  4. 2023Leveraging Large Text Corpora for End-to-End Speech Summarizationcited by 11
  5. 2023Transfer Learning from Pre-trained Language Models Improves End-to-End Speech Summarizationcited by 7
  6. 2024AugSumm: towards generalizable speech summarization using synthetic labels from large language modelcited by 5
  7. 2023BASS: Block-wise Adaptation for Speech Summarizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Exploring Fine-Tuning of Large Audio Language Models for Spoken Language Understanding under Limited Speech Datacited by 6
  2. 2026Towards Multi-Level Transcript Segmentation: LoRA Fine-Tuning for Table-of-Contents Generationcited by 4
  3. 2026Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervisioncited by 1
  4. 2025Advancing Speech Summarization in Multi-modal LLMs with Reinforcement Learningcited by 0