mlsanityData to 14 Oct 2026

CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving

arXiv 2406.10993 ↗First posted 16 Jun 2024cs.CL cs.LG cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

Code-switching is a widely prevalent linguistic phenomenon in multilingual societies like India. Building speech-to-text models for code-switched speech is challenging due to limited availability of datasets. In this work, we focus on the problem of spoken translation (ST) of code-switched speech in Indian languages to English text. We present a new end-to-end model architecture COSTA that scaffolds on pretrained automatic speech recognition (ASR) and machine translation (MT) modules (that are more widely available for many languages). Speech and ASR text representations are fused using an aligned interleaving scheme and are fed further as input to a pretrained MT module; the whole pipeline is then trained end-to-end for spoken translation using synthetically created ST data. We also release a new evaluation benchmark for code-switched Bengali-English, Hindi-English, Marathi-English and Telugu- English speech to English text. COSTA significantly outperforms many competitive cascaded and end-to-end multimodal baselines by up to 3.5 BLEU points.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

37 of 56 references.

References · 37

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  6. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 152
  7. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  8. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  9. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  10. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  11. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  12. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
25 more
  1. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  2. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  3. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  4. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  5. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  6. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  7. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  8. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  9. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  10. 2021Towards Building ASR Systems for the Next Billion Userscited by 47
  11. 2024IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languagescited by 41
  12. 2022IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languagescited by 40
  13. 2020Self-Training for End-to-End Speech Translationcited by 36
  14. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  15. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  16. 2022End-to-End Speech Translation for Code Switched Speechcited by 26
  17. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  18. 2020Self-Supervised Representations Improve End-to-End Speech Translationcited by 21
  19. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
  20. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  21. 2022Code-Switching without Switching: Language Agnostic End-to-End Speech Translationcited by 13
  22. 2018Structured-based Curriculum Learning for End-to-end English-Japanese Speech Translationcited by 11
  23. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  24. 2023Improving speech translation by fusing speech and textcited by 3
  25. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2024Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 14 Indian Languagescited by 5
  2. 2025Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalitiescited by 5
  3. 2025UniCoM: A Universal Code-Switching Speech Generatorcited by 2
  4. 2025Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignmentcited by 1