# VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation

arXiv: https://arxiv.org/abs/2010.16046 · HTML: https://mlsanity.com/paper/2010.16046/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2020.

## Built on by (most cited first, 30 of 34)

- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [A Survey on Cross-Lingual Summarization](https://mlsanity.com/paper/2203.12515/) (2022)
- [Understanding Cross-Lingual Alignment -- A Survey](https://mlsanity.com/paper/2404.06228/) (2024)
- [Enhancing Cross-lingual Transfer by Manifold Mixup](https://mlsanity.com/paper/2205.04182/) (2022)
- [Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation](https://mlsanity.com/paper/2112.03473/) (2021)
- [Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure](https://mlsanity.com/paper/2203.08430/) (2022)
- [Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models](https://mlsanity.com/paper/2210.12360/) (2022)
- [CINO: A Chinese Minority Pre-trained Language Model](https://mlsanity.com/paper/2202.13558/) (2022)
- [DOCmT5: Document-Level Pretraining of Multilingual Language Models](https://mlsanity.com/paper/2112.08709/) (2021)
- [EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching](https://mlsanity.com/paper/2210.12540/) (2022)
- [Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging](https://mlsanity.com/paper/2305.16834/) (2023)
- [T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification](https://mlsanity.com/paper/2306.04996/) (2023)
- [Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models](https://mlsanity.com/paper/2506.13044/) (2025)
- [Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2204.13869/) (2022)
- [ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generation](https://mlsanity.com/paper/2211.04861/) (2022)
- [Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understanding](https://mlsanity.com/paper/2204.07834/) (2022)
- [Synergy with Translation Artifacts for Training and Inference in Multilingual Tasks](https://mlsanity.com/paper/2210.09588/) (2022)
- [Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning](https://mlsanity.com/paper/2304.01295/) (2023)
- [Soft Language Clustering for Multilingual Model Pre-training](https://mlsanity.com/paper/2306.07610/) (2023)
- [Transferable Modeling Strategies for Low-Resource LLM Tasks: A Prompt and Alignment-Based Approach](https://mlsanity.com/paper/2507.00601/) (2025)
- [Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency](https://mlsanity.com/paper/2204.02601/) (2022)
- [OneAligner: Zero-shot Cross-lingual Transfer with One Rich-Resource Language Pair for Low-Resource Sentence Retrieval](https://mlsanity.com/paper/2205.08605/) (2022)
- [Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters](https://mlsanity.com/paper/2205.09470/) (2022)
- [VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning](https://mlsanity.com/paper/2304.08205/) (2023)
- [Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment](https://mlsanity.com/paper/2210.04141/) (2022)
- [mPMR: A Multilingual Pre-trained Machine Reader at Scale](https://mlsanity.com/paper/2305.13645/) (2023)
- [Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching](https://mlsanity.com/paper/2209.05869/) (2022)
- [EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning](https://mlsanity.com/paper/2310.17233/) (2023)
- [Improving Cross-lingual Representation for Semantic Retrieval with Code-switching](https://mlsanity.com/paper/2403.01364/) (2024)

## Abstract (arXiv)

Existing work in multilingual pretraining has demonstrated the potential of cross-lingual transferability by training a unified Transformer encoder for multiple languages. However, much of this work only relies on the shared vocabulary and bilingual contexts to encourage the correlation across languages, which is loose and implicit for aligning the contextual representations between languages. In this paper, we plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. More importantly, when fine-tuning on downstream tasks, the cross-attention module can be plugged in or out on-demand, thus naturally benefiting a wider range of cross-lingual tasks, from language understanding to generation. As a result, the proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark, covering text classification, sequence labeling, question answering, and sentence retrieval. For cross-lingual generation tasks, it also outperforms all existing cross-lingual models and state-of-the-art Transformer variants on WMT14 English-to-German and English-to-French translation datasets, with gains of up to 1~2 BLEU.