mlsanityData to 7 Oct 2026

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

arXiv 2007.07834 ↗First posted 15 Jul 2020cs.CLcited by 237

Not on the map. Why?

Abstract (arXiv)

In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre-training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm.

Key facts. Cited by 237 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 48; 2022: 55; 2023: 49; 2024: 43; 2025: 24; 2026: 18.

237 papers.

Sort:

By area:

  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  2. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  3. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  4. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  5. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  6. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  7. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  8. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  9. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  10. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  11. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  12. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
38 more
  1. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  2. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  3. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  4. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  5. 2022Towards Unsupervised Deep Graph Structure Learningcited by 112
  6. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  7. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  8. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  9. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  10. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  11. 2022XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understandingcited by 68
  12. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 64
  13. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  14. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  15. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  16. 2021MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairscited by 41
  17. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  18. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
  19. 2022VL-BEiT: Generative Vision-Language Pretrainingcited by 39
  20. 2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
  21. 2022A Survey on Cross-Lingual Summarizationcited by 38
  22. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 37
  23. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  24. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  25. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 32
  26. 2022Enhancing Cross-lingual Transfer by Manifold Mixupcited by 30
  27. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 29
  28. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  29. 2022Language Anisotropic Cross-Lingual Model Editingcited by 26
  30. 2022English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddingscited by 26
  31. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 26
  32. 2021Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learningcited by 25
  33. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  34. 2023InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modelingcited by 24
  35. 2023AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languagescited by 24
  36. 2024Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?cited by 23
  37. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22
  38. 2024The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilitiescited by 22

50 most cited of 237 All 237 (JSON)