mlsanityData to 7 Oct 2026

Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

arXiv 1909.00964 ↗First posted 3 Sep 2019cs.CLcited by 96

Not on the map. Why?

Abstract (arXiv)

We present Unicoder, a universal language encoder that is insensitive to different languages. Given an arbitrary NLP task, a model can be trained with Unicoder using training data in one language and directly applied to inputs of the same task in other languages. Comparing to similar efforts such as Multilingual BERT and XLM, three new cross-lingual pre-training tasks are proposed, including cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model. These tasks help Unicoder learn the mappings among different languages from more perspectives. We also find that doing fine-tuning on multiple languages together can bring further improvement. Experiments are performed on two tasks: cross-lingual natural language inference (XNLI) and cross-lingual question answering (XQA), where XLM is our baseline. On XNLI, 1.8% averaged accuracy improvement (on 15 languages) is obtained. On XQA, which is a new cross-lingual dataset built by us, 5.5% averaged accuracy improvement (on French and German) is obtained.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 20; 2023: 18; 2024: 14; 2025: 4; 2026: 2.

96 papers.

Sort:

By area:

  1. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  2. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  3. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  4. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  5. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  6. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  7. 2021Multi-Task Learning in Natural Language Processing: An Overviewcited by 61
  8. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  9. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  10. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Reviewcited by 50
  11. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  12. 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
38 more
  1. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
  2. 2021MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairscited by 41
  3. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  4. 2021Multi-view Subword Regularizationcited by 40
  5. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  6. 2022Enhancing Cross-lingual Transfer by Manifold Mixupcited by 30
  7. 2021Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translationcited by 29
  8. 2023W-MAE: Pre-trained weather model with masked autoencoder for multi-variable weather forecastingcited by 28
  9. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28
  10. 2021Efficient Test Time Adapter Ensembling for Low-resource Language Varietiescited by 26
  11. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  12. 2021Predicting the Performance of Multilingual NLP Modelscited by 24
  13. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  14. 2021Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglotscited by 23
  15. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  16. 2021PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretrainingcited by 21
  17. 2021ProTo: Program-Guided Transformer for Program-Guided Taskscited by 19
  18. 2021XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledgecited by 18
  19. 2022Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structurecited by 17
  20. 2021Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognitioncited by 15
  21. 2022GL-CLeF: A Global-Local Contrastive Learning Framework for Cross-lingual Spoken Language Understandingcited by 14
  22. 2022Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translationcited by 13
  23. 2024German Text Embedding Clustering Benchmarkcited by 13
  24. 2021Cross-lingual Text Classification with Heterogeneous Graph Neural Networkcited by 12
  25. 2021Probing Multilingual Language Models for Discoursecited by 12
  26. 2021SlovakBERT: Slovak Masked Language Modelcited by 11
  27. 2022CINO: A Chinese Minority Pre-trained Language Modelcited by 11
  28. 2024Composing Object Relations and Attributes for Image-Text Matchingcited by 11
  29. 2021X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answeringcited by 10
  30. 2021From Good to Best: Two-Stage Training for Cross-lingual Machine Reading Comprehensioncited by 10
  31. 2023Misalign, Contrast then Distill: Rethinking Misalignments in Language-Image Pretrainingcited by 10
  32. 2024Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Modelcited by 10
  33. 2023Dual-Alignment Pre-training for Cross-lingual Sentence Embeddingcited by 9
  34. 2023BIG-C: a Multimodal Multi-Purpose Dataset for Bembacited by 9
  35. 2025Beyond Literal Token Overlap: Token Alignability for Multilingualitycited by 9
  36. 2021Wine is Not v i n. -- On the Compatibility of Tokenizations Across Languagescited by 8
  37. 2021DOCmT5: Document-Level Pretraining of Multilingual Language Modelscited by 8
  38. 2023T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classificationcited by 8

50 most cited of 96 All 96 (JSON)