# Cross-Lingual Natural Language Generation via Pre-Training

arXiv: https://arxiv.org/abs/1909.10481 · HTML: https://mlsanity.com/paper/1909.10481/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2019.

## Built on by (most cited first, 30 of 63)

- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [XLM-E: Cross-lingual Language Model Pre-training via ELECTRA](https://mlsanity.com/paper/2106.16138/) (2021)
- [Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation](https://mlsanity.com/paper/2205.12647/) (2022)
- [DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders](https://mlsanity.com/paper/2106.13736/) (2021)
- [DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendation](https://mlsanity.com/paper/2109.08877/) (2021)
- [OWL: A Large Language Model for IT Operations](https://mlsanity.com/paper/2309.09298/) (2023)
- [Towards Knowledge-Grounded Counter Narrative Generation for Hate Speech](https://mlsanity.com/paper/2106.11783/) (2021)
- [MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs](https://mlsanity.com/paper/2104.08692/) (2021)
- [Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment](https://mlsanity.com/paper/2106.06381/) (2021)
- [A Survey on Cross-Lingual Summarization](https://mlsanity.com/paper/2203.12515/) (2022)
- [Models and Datasets for Cross-Lingual Summarisation](https://mlsanity.com/paper/2202.09583/) (2022)
- [The Summary Loop: Learning to Write Abstractive Summaries Without Examples](https://mlsanity.com/paper/2105.05361/) (2021)
- [Cross-Lingual Abstractive Summarization with Limited Parallel Resources](https://mlsanity.com/paper/2105.13648/) (2021)
- [ClidSum: A Benchmark Dataset for Cross-Lingual Dialogue Summarization](https://mlsanity.com/paper/2202.05599/) (2022)
- [QAmeleon: Multilingual QA with Only 5 Examples](https://mlsanity.com/paper/2211.08264/) (2022)
- [Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders](https://mlsanity.com/paper/2104.08757/) (2021)
- [ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation](https://mlsanity.com/paper/2106.01597/) (2021)
- [Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training](https://mlsanity.com/paper/2109.07306/) (2021)
- [Distilled Dual-Encoder Model for Vision-Language Understanding](https://mlsanity.com/paper/2112.08723/) (2021)
- [GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator](https://mlsanity.com/paper/2212.10218/) (2022)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Lemur: Log Parsing with Entropy Sampling and Chain-of-Thought Merging](https://mlsanity.com/paper/2402.18205/) (2024)
- [Towards Unifying Multi-Lingual and Cross-Lingual Summarization](https://mlsanity.com/paper/2305.09220/) (2023)
- [mCL-NER: Cross-Lingual Named Entity Recognition via Multi-view Contrastive Learning](https://mlsanity.com/paper/2308.09073/) (2023)
- [PAXQA: Generating Cross-lingual Question Answering Examples at Training Scale](https://mlsanity.com/paper/2304.12206/) (2023)

## Abstract (arXiv)

In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages. We propose to pretrain the encoder and the decoder of a sequence-to-sequence model under both monolingual and cross-lingual settings. The pre-training objective encourages the model to represent different languages in the shared space, so that we can conduct zero-shot cross-lingual transfer. After the pre-training procedure, we use monolingual data to fine-tune the pre-trained model on downstream NLG tasks. Then the sequence-to-sequence model trained in a single language can be directly evaluated beyond that language (i.e., accepting multi-lingual input and producing multi-lingual output). Experimental results on question generation and abstractive summarization show that our model outperforms the machine-translation-based pipeline methods for zero-shot cross-lingual generation. Moreover, cross-lingual transfer improves NLG performance of low-resource languages by leveraging rich-resource language data. Our implementation and data are available at https://github.com/CZWin32768/xnlg.