# Multilingual Pre-training with Language and Task Adaptation for Multilingual Text Style Transfer

arXiv: https://arxiv.org/abs/2203.08552 · HTML: https://mlsanity.com/paper/2203.08552/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 34 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Style Transfer Through Back-Translation](https://mlsanity.com/paper/1804.09000/) (2018)
- [UDapter: Language Adaptation for Truly Universal Dependency Parsing](https://mlsanity.com/paper/2004.14327/) (2020)
- [A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer](https://mlsanity.com/paper/1905.10060/) (2019)
- [XFORMAL: A Benchmark for Multilingual Formality Style Transfer](https://mlsanity.com/paper/2104.04108/) (2021)
- [Expertise Style Transfer: A New Task Towards Better Communication between Experts and Laymen](https://mlsanity.com/paper/2005.00701/) (2020)
- [Parallel Data Augmentation for Formality Style Transfer](https://mlsanity.com/paper/2005.07522/) (2020)
- [Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation](https://mlsanity.com/paper/2004.14911/) (2020)
- [Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer](https://mlsanity.com/paper/2105.06947/) (2021)
- [Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation](https://mlsanity.com/paper/1911.03362/) (2019)
- [Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer](https://mlsanity.com/paper/2110.10668/) (2021)
- [Multilingual Unsupervised Neural Machine Translation with Denoising Adapters](https://mlsanity.com/paper/2110.10472/) (2021)
- [Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters](https://mlsanity.com/paper/2110.09574/) (2021)
- [Reinforced Rewards Framework for Text Style Transfer](https://mlsanity.com/paper/2005.05256/) (2020)
- [Generic resources are what you need: Style transfer tasks without task-specific parallel training data](https://mlsanity.com/paper/2109.04543/) (2021)

## Built on by (most cited first, 8 of 8)

- [Exploring Methods for Cross-lingual Text Style Transfer: The Case of Text Detoxification](https://mlsanity.com/paper/2311.13937/) (2023)
- [mEdIT: Multilingual Text Editing via Instruction Tuning](https://mlsanity.com/paper/2402.16472/) (2024)
- [Multidimensional Evaluation for Text Style Transfer Using ChatGPT](https://mlsanity.com/paper/2304.13462/) (2023)
- [Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation](https://mlsanity.com/paper/2306.00124/) (2023)
- [Exploring Cross-lingual Textual Style Transfer with Large Multilingual Language Models](https://mlsanity.com/paper/2206.02252/) (2022)
- [Event Extraction: A Survey](https://mlsanity.com/paper/2210.03419/) (2022)
- [Multilingual Bias Detection and Mitigation for Indian Languages](https://mlsanity.com/paper/2312.15181/) (2023)
- [UniPSDA: Unsupervised Pseudo Semantic Data Augmentation for Zero-Shot Cross-Lingual Natural Language Understanding](https://mlsanity.com/paper/2406.16372/) (2024)

## Abstract (arXiv)

We exploit the pre-trained seq2seq model mBART for multilingual text style transfer. Using machine translated data as well as gold aligned English sentences yields state-of-the-art results in the three target languages we consider. Besides, in view of the general scarcity of parallel data, we propose a modular approach for multilingual formality transfer, which consists of two training strategies that target adaptation to both language and task. Our approach achieves competitive performance without monolingual task-specific parallel data and can be applied to other style transfer tasks as well as to other languages.