mlsanityData to 7 Oct 2026

Echotune: A Modular Extractor Leveraging the Variable-Length Nature of Speech in ASR Tasks

arXiv 2309.07765 ↗First posted 14 Sep 2023cs.SD cs.CL eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

The Transformer architecture has proven to be highly effective for Automatic Speech Recognition (ASR) tasks, becoming a foundational component for a plethora of research in the domain. Historically, many approaches have leaned on fixed-length attention windows, which becomes problematic for varied speech samples in duration and complexity, leading to data over-smoothing and neglect of essential long-term connectivity. Addressing this limitation, we introduce Echo-MSA, a nimble module equipped with a variable-length attention mechanism that accommodates a range of speech sample complexities and durations. This module offers the flexibility to extract speech features across various granularities, spanning from frames and phonemes to words and discourse. The proposed design captures the variable length feature of speech and addresses the limitations of fixed-length attention. Our evaluation leverages a parallel attention architecture complemented by a dynamic gating mechanism that amalgamates traditional attention with the Echo-MSA module output. Empirical evidence from our study reveals that integrating Echo-MSA into the primary model's training regime significantly enhances the word error rate (WER) performance, all while preserving the intrinsic stability of the original model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

18 of 31 references.

References · 18

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  6. 2015Listen, Attend and Spellcited by 542
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  8. 2015Skip-Thought Vectorscited by 321
  9. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  10. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  11. 2016Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Fillingcited by 137
  12. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 81
6 more
  1. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  2. 2020SlimIPL: Language-Model-Free Iterative Pseudo-Labelingcited by 54
  3. 2019Neural source-filter waveform models for statistical parametric speech synthesiscited by 46
  4. 2023Transformers in Speech Processing: A Surveycited by 38
  5. 2016Very Deep Convolutional Networks for End-to-End Speech Recognitioncited by 34
  6. 2017A Study of All-Convolutional Encoders for Connectionist Temporal Classificationcited by 3

Built on by

Not cited yet.