# Mixed-Lingual Pre-training for Cross-lingual Summarization

arXiv: https://arxiv.org/abs/2010.08892 · HTML: https://mlsanity.com/paper/2010.08892/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2020.

## Built on by (most cited first, 17 of 17)

- [A Survey on Cross-Lingual Summarization](https://mlsanity.com/paper/2203.12515/) (2022)
- [ClidSum: A Benchmark Dataset for Cross-Lingual Dialogue Summarization](https://mlsanity.com/paper/2202.05599/) (2022)
- [A Variational Hierarchical Model for Neural Cross-Lingual Summarization](https://mlsanity.com/paper/2203.03820/) (2022)
- [Zero-Shot Cross-Lingual Summarization via Large Language Models](https://mlsanity.com/paper/2302.14229/) (2023)
- [Towards Unifying Multi-Lingual and Cross-Lingual Summarization](https://mlsanity.com/paper/2305.09220/) (2023)
- [Understanding Translationese in Cross-Lingual Summarization](https://mlsanity.com/paper/2212.07220/) (2022)
- [The Cross-lingual Conversation Summarization Challenge](https://mlsanity.com/paper/2205.00379/) (2022)
- [Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation](https://mlsanity.com/paper/2307.04018/) (2023)
- [Long-Document Cross-Lingual Summarization](https://mlsanity.com/paper/2212.00586/) (2022)
- [Unifying Cross-lingual Summarization and Machine Translation with Compression Rate](https://mlsanity.com/paper/2110.07936/) (2021)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation](https://mlsanity.com/paper/2305.09652/) (2023)
- [ClueGraphSum: Let Key Clues Guide the Cross-Lingual Abstractive Summarization](https://mlsanity.com/paper/2203.02797/) (2022)
- [VieSum: How Robust Are Transformer-based Models on Vietnamese Summarization?](https://mlsanity.com/paper/2110.04257/) (2021)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [SimCSum: Joint Learning of Simplification and Cross-lingual Summarization for Cross-lingual Science Journalism](https://mlsanity.com/paper/2304.01621/) (2023)
- [An Empirical Study of Many-to-Many Summarization with Large Language Models](https://mlsanity.com/paper/2505.12983/) (2025)

## Abstract (arXiv)

Cross-lingual Summarization (CLS) aims at producing a summary in the target language for an article in the source language. Traditional solutions employ a two-step approach, i.e. translate then summarize or summarize then translate. Recently, end-to-end models have achieved better results, but these approaches are mostly limited by their dependence on large-scale labeled data. We propose a solution based on mixed-lingual pre-training that leverages both cross-lingual tasks such as translation and monolingual tasks like masked language models. Thus, our model can leverage the massive monolingual data to enhance its modeling of language. Moreover, the architecture has no task-specific components, which saves memory and increases optimization efficiency. We show in experiments that this pre-training scheme can effectively boost the performance of cross-lingual summarization. In Neural Cross-Lingual Summarization (NCLS) dataset, our model achieves an improvement of 2.82 (English to Chinese) and 1.15 (Chinese to English) ROUGE-1 scores over state-of-the-art results.