Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Models
Not on the map. Why?
Abstract (arXiv)
Spoken language models (SLMs) that integrate speech with large language models (LMs) rely on modality adapters (MAs) to map the output of speech encoders to a representation that is understandable to the decoder LM. Yet we know very little about how these crucial MAs transform representations. Here we examine the MA output representation in three SLMs (SALMONN, Qwen2-Audio and Phi-4-Multimodal-Instruct). By finding the nearest decoder LM token to an MA representation, we uncover two strategies for MA representations. For models using a Whisper encoder, MAs appear to represent the meaning of the input using an English-based interlingua, allowing them to handle languages unseen in instruction tuning. For models that don't, like Phi-4-Multimodal-Instruct, MAs instead represent the phonetics of the input, but expressed with English words. We hypothesise that which arises depends on whether the speech encoder is trained only for speech recognition or also for translation.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 2 Oct 2025.
Builds on
27 of 34 references.
References · 27
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2024Qwen2 Technical Reportcited by 2,076
- 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
- 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
- 2024Qwen2-Audio Technical Reportcited by 543
- 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
- 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
- 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
15 more
- 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
- 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
- 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
- 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
- 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
- 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
- 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
- 2023Joint Audio and Speech Understandingcited by 99
- 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
- 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
- 2022Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?cited by 27
- 2020Similarity Analysis of Self-Supervised Speech Representationscited by 26
- 2021Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processingcited by 26
- 2023An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speechcited by 15
- 2022Quantifying Language Variation Acoustically with Few Resourcescited by 12
Built on by
Citing papers per year: 2026: 2.
2 papers.