mlsanityData to 7 Oct 2026

Beyond Universal Transformer: block reusing with adaptor in Transformer for automatic speech recognition

arXiv 2303.13072 ↗First posted 23 Mar 2023cs.SD cs.CL eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Transformer-based models have recently made significant achievements in the application of end-to-end (E2E) automatic speech recognition (ASR). It is possible to deploy the E2E ASR system on smart devices with the help of Transformer-based models. While these models still have the disadvantage of requiring a large number of model parameters. To overcome the drawback of universal Transformer models for the application of ASR on edge devices, we propose a solution that can reuse the block in Transformer models for the occasion of the small footprint ASR system, which meets the objective of accommodating resource limitations without compromising recognition accuracy. Specifically, we design a novel block-reusing strategy for speech Transformer (BRST) to enhance the effectiveness of parameters and propose an adapter module (ADM) that can produce a compact and adaptable model with only a few additional trainable parameters accompanying each reusing block. We conducted an experiment with the proposed method on the public AISHELL-1 corpus, and the results show that the proposed approach achieves the character error rate (CER) of 9.3%/6.63% with only 7.6M/8.3M parameters without and with the ADM, respectively. In addition, we also make a deeper analysis to show the effect of ADM in the general block-reusing method.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

18 of 29 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2018Universal Transformerscited by 613
  9. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  10. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  11. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  12. 2019SpecAugment on Large Scale Datasetscited by 60
6 more
  1. 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
  2. 2021Pushing the Limits of Non-Autoregressive Speech Recognitioncited by 18
  3. 2020PyChain: A Fully Parallelized PyTorch Implementation of LF-MMI for End-to-End ASRcited by 16
  4. 2021Of Non-Linearity and Commutativity in BERTcited by 16
  5. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Modelscited by 10
  6. 2021Extremely Low Footprint End-to-End ASR System for Smart Devicecited by 7

Built on by

Not cited yet.