Data to 7 Oct 2026

fairseq S2T: Fast Speech-to-Text Modeling with fairseq

arXiv 2010.05171 ↗First posted 11 Oct 2020cs.CL eess.AScited by 174

Not on the map. Why?

Abstract (arXiv)

We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation. It follows fairseq's careful design for scalability and extensibility. We provide end-to-end workflows from data pre-processing, model training to offline (online) inference. We implement state-of-the-art RNN-based, Transformer-based as well as Conformer-based models and open-source detailed training recipes. Fairseq's machine translation models and language models can be seamlessly integrated into S2T workflows for multi-task learning or transfer learning. Fairseq S2T documentation and examples are available at https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text.

Key facts. Cited by 174 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 49; 2023: 42; 2024: 26; 2025: 23; 2026: 3.

174 papers.

Sort:

By area:

2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
2021Direct speech-to-speech translation with discrete unitscited by 148
2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
2021Textless Speech-to-Speech Translation on Real Datacited by 121
2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
2022Cross-modal Contrastive Learning for Speech Translationcited by 61
2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57

38 more

2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 46
2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
2022Information-Transport-based Policy for Simultaneous Translationcited by 40
2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 39
2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
2023Transformers in Speech Processing: A Surveycited by 38
2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
2022Attention as a Guide for Simultaneous Speech Translationcited by 32
2021Learning When to Translate for Streaming Speechcited by 30
2022Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speechcited by 30
2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26
2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
2021Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Taskscited by 24
2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
2022Does Simultaneous Speech Translation need Simultaneous Models?cited by 22
2022SDS-200: A Swiss German Speech to Standard German Text Corpuscited by 22
2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
2023End-to-End Simultaneous Speech Translation with Differentiable Segmentationcited by 22
2024Benchmarking Large Multimodal Models against Common Corruptionscited by 22
2022M3ST: Mix at Three Levels for Speech Translationcited by 21
2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
2024Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learningcited by 20
2022TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalitiescited by 19
2024StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selectioncited by 19
2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
2023Zambezi Voice: A Multilingual Speech Corpus for Zambian Languagescited by 18
2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
2021End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021cited by 16

50 most cited of 174 All 174 (JSON)