Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation
Not on the map. Why?
Abstract (arXiv)
This paper introduces a novel approach called sentence-wise speech summarization (Sen-SSum), which generates text summaries from a spoken document in a sentence-by-sentence manner. Sen-SSum combines the real-time processing of automatic speech recognition (ASR) with the conciseness of speech summarization. To explore this approach, we present two datasets for Sen-SSum: Mega-SSum and CSJ-SSum. Using these datasets, our study evaluates two types of Transformer-based models: 1) cascade models that combine ASR and strong text summarization models, and 2) end-to-end (E2E) models that directly convert speech into a text summary. While E2E models are appealing to develop compute-efficient models, they perform worse than cascade models. Therefore, we propose knowledge distillation for E2E models using pseudo-summaries generated by the cascade models. Our experiments show that this proposed knowledge distillation effectively improves the performance of the E2E model on both datasets.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 1 Aug 2024.
Builds on
28 of 40 references.
References · 28
- 2017Attention Is All You Needcited by 59,932
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2016Sequence-Level Knowledge Distillationcited by 761
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
16 more
- 2023End-to-End Speech Recognition: A Surveycited by 127
- 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 100
- 2017Neural Machine Translation with Extended Contextcited by 95
- 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
- 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
- 2021Speech Summarization using Restricted Self-Attentioncited by 23
- 2020Improving Readability for Automatic Speech Recognition Transcriptioncited by 22
- 2021Neural Inverse Text Normalizationcited by 19
- 2018Disfluency Detection using a Noisy Channel Model and a Deep Neural Language Modelcited by 11
- 2023Leveraging Large Text Corpora for End-to-End Speech Summarizationcited by 11
- 2020Adapting End-to-End Speech Recognition for Readable Subtitlescited by 10
- 2021Attention-based Multi-hypothesis Fusion for Speech Summarizationcited by 9
- 2022Abstractive Meeting Summarization: A Surveycited by 9
- 2023SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?cited by 9
- 2023Transfer Learning from Pre-trained Language Models Improves End-to-End Speech Summarizationcited by 7
- 2022Streaming Joint Speech Recognition and Disfluency Detectioncited by 6
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.