mlsanityData to 7 Oct 2026

Speech Summarization using Restricted Self-Attention

arXiv 2110.06263 ↗First posted 12 Oct 2021cs.CL cs.AI cs.SDcited by 23

Not on the map. Why?

Abstract (arXiv)

Speech summarization is typically performed by using a cascade of speech recognition and text summarization models. End-to-end modeling of speech summarization models is challenging due to memory and compute constraints arising from long input audio sequences. Recent work in document summarization has inspired methods to reduce the complexity of self-attentions, which enables transformer models to handle long sequences. In this work, we introduce a single model optimized end-to-end for speech summarization. We apply the restricted self-attention technique from text-based models to speech models to address the memory and compute constraints. We demonstrate that the proposed model learns to directly summarize speech for the How-2 corpus of instructional videos. The proposed end-to-end model outperforms the previously proposed cascaded model by 3 points absolute on ROUGE. Further, we consider the spoken language understanding task of predicting concepts from speech inputs and show that the proposed end-to-end model outperforms the cascade model by 4 points absolute F-1.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

19 of 29 references.

References · 19

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  10. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  11. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  12. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
7 more
  1. 2019Joint Speech Recognition and Speaker Diarization via Sequence Transductioncited by 49
  2. 2019Multimodal Abstractive Summarization for How2 Videoscited by 41
  3. 2021Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarizationcited by 32
  4. 2021Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformerscited by 25
  5. 2021Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillationcited by 12
  6. 2020Automatic Speech Summarisation: A Scoping Reviewcited by 6
  7. 2021Capturing Multi-Resolution Context by Dilated Self-Attentioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 6; 2024: 9; 2025: 2; 2026: 2.

23 papers.

Sort:

By area:

  1. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 34
  2. 2024Exploring the Capability of Mamba in Speech Applicationscited by 19
  3. 2023Leveraging Large Text Corpora for End-to-End Speech Summarizationcited by 11
  4. 2024A Survey of Video Datasets for Grounded Event Understandingcited by 9
  5. 2024On the Evaluation of Speech Foundation Models for Spoken Language Understandingcited by 9
  6. 2022Assessing ASR Model Quality on Disordered Speech using BERTScorecited by 8
  7. 2022Context-aware Fine-tuning of Self-supervised Speech Modelscited by 7
  8. 2023Transfer Learning from Pre-trained Language Models Improves End-to-End Speech Summarizationcited by 7
  9. 2023ed-cec: improving rare word recognition using asr postprocessing based on error detection and context-aware error correctioncited by 7
  10. 2024Prompting Large Language Models with Audio for General-Purpose Speech Summarizationcited by 6
  11. 2024AugSumm: towards generalizable speech summarization using synthetic labels from large language modelcited by 5
  12. 2025Summarizing Speech: A Comprehensive Surveycited by 5
11 more
  1. 2023BASS: Block-wise Adaptation for Speech Summarizationcited by 4
  2. 2024An End-to-End Speech Summarization Using Large Language Modelcited by 4
  3. 2026Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarizationcited by 4
  4. 2024Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillationcited by 3
  5. 2024Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?cited by 3
  6. 2022XNOR-FORMER: Learning Accurate Approximations in Long Speech Transformerscited by 2
  7. 2023Factual Consistency Oriented Speech Recognitioncited by 1
  8. 2023Towards End-to-end Speech-to-text Summarizationcited by 0
  9. 2024Bridging the Gap between Text, Audio, Image, and Any Sequence: A Novel Approach using Gloss-based Annotationcited by 0
  10. 2025Advancing Speech Summarization in Multi-modal LLMs with Reinforcement Learningcited by 0
  11. 2026Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarizationcited by 0