mlsanityData to 14 Oct 2026

Bridging the Granularity Gap for Acoustic Modeling

arXiv 2305.17356 ↗First posted 27 May 2023cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

While Transformer has become the de-facto standard for speech, modeling upon the fine-grained frame-level features remains an open challenge of capturing long-distance dependencies and distributing the attention weights. We propose textitProgressive Down-Sampling (PDS) which gradually compresses the acoustic features into coarser-grained units containing more complete semantic information, like text-level representation. In addition, we develop a representation fusion method to alleviate information loss that occurs inevitably during high compression. In this way, we compress the acoustic features into 1/32 of the initial length while achieving better or comparable performances on the speech recognition task. And as a bonus, it yields inference speedups ranging from 1.20× to 1.47×. By reducing the modeling burden, we also achieve competitive results when training on the more challenging speech translation task.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

34 of 52 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2016Pyramid Scene Parsing Networkcited by 2,479
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  8. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  9. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  10. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  11. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 151
  12. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
22 more
  1. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  2. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 97
  3. 2020Feature Pyramid Transformercited by 96
  4. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  5. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  6. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  7. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  8. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  9. 2019Very Deep Self-Attention Networks for End-to-End Speech Recognitioncited by 46
  10. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  11. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  12. 2018Self-Attentional Acoustic Modelscited by 29
  13. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  14. 2020Adaptive Feature Selection for End-to-End Speech Translationcited by 25
  15. 2020Exploring Transformers for Large-Scale Speech Recognitioncited by 24
  16. 2020Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognitioncited by 23
  17. 2019Exploring Phoneme-Level Speech Representations for End-to-End Speech Translationcited by 15
  18. 2021Speechformer: Reducing Information Loss in Direct Speech Translationcited by 15
  19. 2020Phone Features Improve Speech Translationcited by 11
  20. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9
  21. 2021Efficient Transformer for Direct Speech Translationcited by 5
  22. 2022Uconv-Conformer: High Reduction of Input Sequence Length for End-to-End Speech Recognitioncited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  2. 2023Soft Alignment of Modality Space for End-to-end Speech Translationcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models
  2. 2026X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
  3. 2026FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation
  4. 2022ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech
  5. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
  6. 2026TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech
  7. 2025Towards Audio Token Compression in Large Audio Language Models
  8. 2022Variable-rate hierarchical CPC leads to acoustic unit discovery in speech