mlsanityData to 14 Oct 2026

Similarity and Content-based Phonetic Self Attention for Speech Recognition

arXiv 2203.10252 ↗First posted 19 Mar 2022cs.CL eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

Transformer-based speech recognition models have achieved great success due to the self-attention (SA) mechanism that utilizes every frame in the feature extraction process. Especially, SA heads in lower layers capture various phonetic characteristics by the query-key dot product, which is designed to compute the pairwise relationship between frames. In this paper, we propose a variant of SA to extract more representative phonetic features. The proposed phonetic self-attention (phSA) is composed of two different types of phonetic attention; one is similarity-based and the other is content-based. In short, similarity-based attention captures the correlation between frames while content-based attention only considers each frame without being affected by other frames. We identify which parts of the original dot product equation are related to two different attention patterns and improve each part with simple modifications. Our experiments on phoneme classification and speech recognition show that replacing SA with phSA for lower layers improves the recognition performance without increasing the latency and the parameter size.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 24 of its 29 references that are in the map. First posted on arXiv 19 Mar 2022.

Builds on

24 of 29 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  7. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  8. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  9. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  10. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  11. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  12. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
12 more
  1. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 77
  2. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  3. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  4. 2020On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformerscited by 24
  5. 2020Relative Positional Encoding for Speech Recognition and Direct Translationcited by 23
  6. 2020Simplified Self-Attention for Transformer-based End-to-End Speech Recognitioncited by 20
  7. 2021Pushing the Limits of Non-Autoregressive Speech Recognitioncited by 18
  8. 2020Understanding Self-Attention of Self-Supervised Audio Transformerscited by 15
  9. 2021Aligned Contrastive Predictive Codingcited by 15
  10. 2019BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognitioncited by 10
  11. 2021Efficient Conformer with Prob-Sparse Attention Mechanism for End-to-EndSpeech Recognitioncited by 9
  12. 2020Transformer-based End-to-End Speech Recognition with Local Dense Synthesizer Attentioncited by 6

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 0; 2025: 1.

3 papers.

  1. 2023Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layercited by 2
  2. 2025Dynamic Acoustic Model Architecture Optimization in Training for ASRcited by 1
  3. 2022A Comparison of Transformer, Convolutional, and Recurrent Neural Networks on Phoneme Recognitioncited by 0