# Relative Positional Encoding for Speech Recognition and Direct Translation

arXiv: https://arxiv.org/abs/2005.09940 · HTML: https://mlsanity.com/paper/2005.09940/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2020.

## Built on by (most cited first, 23 of 23)

- [The Multilingual TEDx Corpus for Speech Recognition and Translation](https://mlsanity.com/paper/2102.01757/) (2021)
- [From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformers](https://mlsanity.com/paper/2107.07999/) (2021)
- [Relative Positional Encoding for Transformers with Linear Complexity](https://mlsanity.com/paper/2105.08399/) (2021)
- [CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022](https://mlsanity.com/paper/2204.06028/) (2022)
- [Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videos](https://mlsanity.com/paper/2206.04523/) (2022)
- [Variable Attention Masking for Configurable Transformer Transducer Speech Recognition](https://mlsanity.com/paper/2211.01438/) (2022)
- [AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation](https://mlsanity.com/paper/2312.02512/) (2023)
- [Efficient Weight factorization for Multilingual Speech Recognition](https://mlsanity.com/paper/2105.03010/) (2021)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates](https://mlsanity.com/paper/2109.12804/) (2021)
- [Adaptive multilingual speech recognition with pretrained models](https://mlsanity.com/paper/2205.12304/) (2022)
- [Accent conversion using discrete units with parallel data synthesized from controllable accented TTS](https://mlsanity.com/paper/2410.03734/) (2024)
- [Improving Pronunciation and Accent Conversion through Knowledge Distillation And Synthetic Ground-Truth from Native TTS](https://mlsanity.com/paper/2410.14997/) (2024)
- [Transformer-based end-to-end speech recognition with residual Gaussian-based self-attention](https://mlsanity.com/paper/2103.15722/) (2021)
- [Conformer-based End-to-end Speech Recognition With Rotary Position Embedding](https://mlsanity.com/paper/2107.05907/) (2021)
- [Similarity and Content-based Phonetic Self Attention for Speech Recognition](https://mlsanity.com/paper/2203.10252/) (2022)
- [Automatic speech recognition for the Nepali language using CNN, bidirectional LSTM and ResNet](https://mlsanity.com/paper/2406.17825/) (2024)
- [A Circular Argument : Does RoPE need to be Equivariant for Vision?](https://mlsanity.com/paper/2511.08368/) (2025)
- [Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms](https://mlsanity.com/paper/2105.05791/) (2021)
- [The NiuTrans End-to-End Speech Translation System for IWSLT 2021 Offline Task](https://mlsanity.com/paper/2107.02444/) (2021)
- [Streaming Non-Autoregressive Model for Accent Conversion and Pronunciation Improvement](https://mlsanity.com/paper/2506.16580/) (2025)
- [ESPnet-ST IWSLT 2021 Offline Speech Translation System](https://mlsanity.com/paper/2107.00636/) (2021)
- [The IWSLT 2021 BUT Speech Translation Systems](https://mlsanity.com/paper/2107.06155/) (2021)

## Abstract (arXiv)

Transformer models are powerful sequence-to-sequence architectures that are capable of directly mapping speech inputs to transcriptions or translations. However, the mechanism for modeling positions in this model was tailored for text modeling, and thus is less ideal for acoustic inputs. In this work, we adapt the relative position encoding scheme to the Speech Transformer, where the key addition is relative distance between input states in the self-attention network. As a result, the network can better adapt to the variable distributions present in speech data. Our experiments show that our resulting model achieves the best recognition result on the Switchboard benchmark in the non-augmentation condition, and the best published result in the MuST-C speech translation benchmark. We also show that this model is able to better utilize synthetic data than the Transformer, and adapts better to variable sentence segmentation quality for speech translation.