mlsanityData to 14 Oct 2026

Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarios

arXiv 2505.24691 ↗First posted 30 May 2025cs.CL cs.SD eess.AScited by 3

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

We propose a Speech-to-Text Translation (S2TT) approach that integrates phoneme representations into a Chain-of-Thought (CoT) framework to improve translation in low-resource and zero-resource settings. By introducing phoneme recognition as an intermediate step, we enhance cross-lingual transfer, enabling translation even for languages with no labeled speech data. Our system builds on a multilingual LLM, which we extend to process speech and phonemes. Training follows a curriculum learning strategy that progressively introduces more complex tasks. Experiments on multilingual S2TT benchmarks show that phoneme-augmented CoT improves translation quality in low-resource conditions and enables zero-resource translation, while slightly impacting high-resource performance. Despite this trade-off, our findings demonstrate that phoneme-based CoT is a promising step toward making S2TT more accessible across diverse languages.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 33 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

26 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Chain-of-Thought Prompting for Speech Translation17 Sep
  2. 2024Investigating Decoder-only Large Language Models for Speech-to-text Translation3 Jul

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  4. 2023PaLM 2 Technical Reportcited by 1,162
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  6. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  7. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  8. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  9. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  10. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  11. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  12. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
12 more
  1. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  2. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  3. 2023Textually Pretrained Speech Language Modelscited by 101
  4. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  5. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  6. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  7. 2023Speech Translation with Large Language Models: An Industrial Practicecited by 28
  8. 2024Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learningcited by 20
  9. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
  10. 2024Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?cited by 13
  11. 2025Salamandra Technical Reportcited by 10
  12. 2024Zero-resource Speech Translation and Recognition with LLMscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025POWSM: A Phonetic Open Whisper-Style Speech Foundation Modelcited by 14
  2. 2025Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translationcited by 1
  3. 2026Transcribe, Translate, and Optimize: Joint Reward Learning for Speech Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?
  2. 2026Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
  3. 2025Improving Language and Modality Transfer in Translation by Character-level Modeling
  4. 2023Improved Cross-Lingual Transfer Learning For Automatic Speech Translation
  5. 2024Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language
  6. 2024Refining Translations with LLMs: A Constraint-Aware Iterative Prompting Approach
  7. 2024Mufu: Multilingual Fused Learning for Low-Resource Translation with LLM
  8. 2025How and Where to Translate? The Impact of Translation Strategies in Cross-lingual LLM Prompting