mlsanityData to 7 Oct 2026

On the limited utility of parallel data for learning shared multilingual representations

arXiv 2603.29026 ↗First posted 30 Mar 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Shared multilingual representations are essential for cross-lingual tasks and knowledge transfer across languages. This study looks at the impact of parallel data, i.e. translated sentences, in pretraining as a signal to trigger representations that are aligned across languages. We train reference models with different proportions of parallel data and show that parallel data seem to have only a minimal effect on the cross-lingual alignment. Based on multiple evaluation methods, we find that the effect is limited to potentially accelerating the representation sharing in the early phases of pretraining, and to decreasing the amount of language-specific neurons in the model. Cross-lingual alignment seems to emerge on similar levels even without the explicit signal from parallel data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

22 of 37 references.

References · 22

  1. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  7. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  9. 2019How multilingual is Multilingual BERT?cited by 642
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  11. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  12. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
10 more
  1. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  2. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  3. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  4. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
  5. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  6. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  7. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  8. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  9. 2021nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?cited by 21
  10. 2024Poro 34B and the Blessing of Multilingualitycited by 18

Built on by

Not cited yet.