mlsanityData to 14 Oct 2026

How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations

arXiv 2411.17666 ↗First posted 26 Nov 2024cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Multimodal foundation models aim to create a unified representation space that abstracts away from surface features like language syntax or modality differences. To investigate this, we study the internal representations of three recent models, analyzing the model activations from semantically equivalent sentences across languages in the text and speech modalities. Our findings reveal that: 1) Cross-modal representations converge over model layers, except in the initial layers specialized at text and speech processing. 2) Length adaptation is crucial for reducing the cross-modal gap between text and speech, although current approaches' effectiveness is primarily limited to high-resource languages. 3) Speech exhibits larger cross-lingual differences than text. 4) For models not explicitly trained for modality-agnostic representations, the modality gap is more prominent than the language gap.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

32 of 46 references.

References · 32

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2018Deep contextualized word representationscited by 2,290
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  7. 2023Large Language Modelscited by 780
  8. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  9. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  10. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  11. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  12. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
20 more
  1. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  2. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  3. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  4. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  5. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 108
  6. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  7. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  8. 2023Analysing Discrete Self Supervised Speech Representation for Spoken Language Modelingcited by 47
  9. 2021CTC-based Compression for Direct Speech Translationcited by 45
  10. 2019Multilingual End-to-End Speech Translationcited by 37
  11. 2020Improving Zero-Shot Translation by Disentangling Positional Informationcited by 35
  12. 2022Probing phoneme, language and speaker information in unsupervised speech representationscited by 33
  13. 2017Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systemscited by 32
  14. 2024Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Modelscited by 23
  15. 2023Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfercited by 22
  16. 2022How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?cited by 20
  17. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  18. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  19. 2022Tackling data scarcity in speech translation using zero-shot multilingual machine translation techniquescited by 4
  20. 2024Speech Representation Analysis based on Inter- and Intra-Model Similaritiescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  2. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 11
  3. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 2
  4. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Modelscited by 1
  5. 2025Leveraging Intermediate Representations of Time Series Foundation Models for Anomaly Detectioncited by 1
  6. 2026The Alignment Curse: Modality Alignment Supercharges Audio Attacks via Text Transfercited by 1
  7. 2026Generation-Step-Aware Framework for Cross-Modal Representation and Control in Multilingual Speech-Text Modelscited by 0
  8. 2026Multilinguality in Hybrid Attention LLMscited by 0
  9. 2026Architecture-Dependent Fusion Pathways in MLLMscited by 0