# "Mask and Infill" : Applying Masked Language Model to Sentiment Transfer

arXiv: https://arxiv.org/abs/1908.08039 · HTML: https://mlsanity.com/paper/1908.08039/

## Key facts

- Cited by 48 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 21 Aug 2019.

## Built on by (most cited first, 30 of 48)

- [Text Detoxification using Large Pre-trained Neural Models](https://mlsanity.com/paper/2109.08914/) (2021)
- [Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models](https://mlsanity.com/paper/2205.11503/) (2022)
- [LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer](https://mlsanity.com/paper/2105.08206/) (2021)
- [Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts](https://mlsanity.com/paper/2212.10543/) (2022)
- [StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer](https://mlsanity.com/paper/2104.05196/) (2021)
- [Civil Rephrases Of Toxic Texts With Self-Supervised Transformers](https://mlsanity.com/paper/2102.05456/) (2021)
- [Text Style Transfer for Bias Mitigation using Masked Language Modeling](https://mlsanity.com/paper/2201.08643/) (2022)
- [Don't lose the message while paraphrasing: A study on content preserving style transfer](https://mlsanity.com/paper/2308.09055/) (2023)
- [A Survey on Natural Language Counterfactual Generation](https://mlsanity.com/paper/2407.03993/) (2024)
- [Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization](https://mlsanity.com/paper/2108.00449/) (2021)
- [Think Twice: A Human-like Two-stage Conversational Agent for Emotional Response Generation](https://mlsanity.com/paper/2301.04907/) (2023)
- [STEER: Unified Style Transfer with Expert Reinforcement](https://mlsanity.com/paper/2311.07167/) (2023)
- [Methods for Detoxification of Texts for the Russian Language](https://mlsanity.com/paper/2105.09052/) (2021)
- [Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks](https://mlsanity.com/paper/2202.13840/) (2022)
- [A Call for Standardization and Validation of Text Style Transfer Evaluation](https://mlsanity.com/paper/2306.00539/) (2023)
- [Transductive Learning for Unsupervised Text Style Transfer](https://mlsanity.com/paper/2109.07812/) (2021)
- [Distilling Text Style Transfer With Self-Explanation From LLMs](https://mlsanity.com/paper/2403.01106/) (2024)
- [Transferable Persona-Grounded Dialogues via Grounded Minimal Edits](https://mlsanity.com/paper/2109.07713/) (2021)
- [Making a (Counterfactual) Difference One Rationale at a Time](https://mlsanity.com/paper/2201.05177/) (2022)
- [StoryTrans: Non-Parallel Story Author-Style Transfer with Discourse Representations and Content Enhancing](https://mlsanity.com/paper/2208.13423/) (2022)
- [Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews](https://mlsanity.com/paper/2104.05861/) (2021)
- [NAST: A Non-Autoregressive Generator with Word Alignment for Unsupervised Text Style Transfer](https://mlsanity.com/paper/2106.02210/) (2021)
- [Efficient Reinforcement Learning for Unsupervised Controlled Text Generation](https://mlsanity.com/paper/2204.07696/) (2022)
- [A Review of Human Emotion Synthesis Based on Generative Technology](https://mlsanity.com/paper/2412.07116/) (2024)
- [DiffuDetox: A Mixed Diffusion Model for Text Detoxification](https://mlsanity.com/paper/2306.08505/) (2023)
- [From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Survey](https://mlsanity.com/paper/2110.15871/) (2021)
- [LMStyle Benchmark: Evaluating Text Style Transfer for Chatbots](https://mlsanity.com/paper/2403.08943/) (2024)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Domain Confused Contrastive Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2207.04564/) (2022)
- [(Psycho-)Linguistic Features Meet Transformer Models for Improved Explainable and Controllable Text Simplification](https://mlsanity.com/paper/2212.09848/) (2022)

## Abstract (arXiv)

This paper focuses on the task of sentiment transfer on non-parallel text, which modifies sentiment attributes (e.g., positive or negative) of sentences while preserving their attribute-independent content. Due to the limited capability of RNNbased encoder-decoder structure to capture deep and long-range dependencies among words, previous works can hardly generate satisfactory sentences from scratch. When humans convert the sentiment attribute of a sentence, a simple but effective approach is to only replace the original sentimental tokens in the sentence with target sentimental expressions, instead of building a new sentence from scratch. Such a process is very similar to the task of Text Infilling or Cloze, which could be handled by a deep bidirectional Masked Language Model (e.g. BERT). So we propose a two step approach "Mask and Infill". In the mask step, we separate style from content by masking the positions of sentimental tokens. In the infill step, we retrofit MLM to Attribute Conditional MLM, to infill the masked positions by predicting words or phrases conditioned on the context1 and target sentiment. We evaluate our model on two review datasets with quantitative, qualitative, and human evaluations. Experimental results demonstrate that our models improve state-of-the-art performance.