# Improving Formality Style Transfer with Context-Aware Rule Injection

arXiv: https://arxiv.org/abs/2106.00210 · HTML: https://mlsanity.com/paper/2106.00210/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 1 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [Generating Sentences from Disentangled Syntactic and Semantic Spaces](https://mlsanity.com/paper/1907.05789/) (2019)
- [Shakespearizing Modern Language Using Copy-Enriched Sequence-to-Sequence Models](https://mlsanity.com/paper/1707.01161/) (2017)
- [Low-Memory Neural Network Training: A Technical Report](https://mlsanity.com/paper/1904.10631/) (2019)
- [Controllable Paraphrase Generation with a Syntactic Exemplar](https://mlsanity.com/paper/1906.00565/) (2019)
- [Parallel Data Augmentation for Formality Style Transfer](https://mlsanity.com/paper/2005.07522/) (2020)
- [Multi-Task Neural Models for Translating Between Styles Within and Across Languages](https://mlsanity.com/paper/1806.04357/) (2018)
- [Formality Style Transfer with Hybrid Textual Annotations](https://mlsanity.com/paper/1903.06353/) (2019)
- [Zero-shot Entity Linking with Efficient Long Range Sequence Modeling](https://mlsanity.com/paper/2010.06065/) (2020)
- [MoNoise: Modeling Noise Using a Modular Normalization System](https://mlsanity.com/paper/1710.03476/) (2017)
- [SeerNet at SemEval-2018 Task 1: Domain Adaptation for Affect in Tweets](https://mlsanity.com/paper/1804.06137/) (2018)

## Built on by (most cited first, 13 of 13)

- [NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notes](https://mlsanity.com/paper/2310.15959/) (2023)
- [Editing Personality for Large Language Models](https://mlsanity.com/paper/2310.02168/) (2023)
- [Extracting Biomedical Factual Knowledge Using Pretrained Language Model and Electronic Health Record Context](https://mlsanity.com/paper/2209.07859/) (2022)
- [Context Variance Evaluation of Pretrained Language Models for Prompt-based Biomedical Knowledge Probing](https://mlsanity.com/paper/2211.10265/) (2022)
- [UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations?](https://mlsanity.com/paper/2306.16931/) (2023)
- [Semi-Supervised Formality Style Transfer with Consistency Training](https://mlsanity.com/paper/2203.13620/) (2022)
- [ReadCtrl: Personalizing text generation with readability-controlled instruction learning](https://mlsanity.com/paper/2406.09205/) (2024)
- [Automated Identification of Eviction Status from Electronic Health Record Notes](https://mlsanity.com/paper/2212.02762/) (2022)
- [From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Survey](https://mlsanity.com/paper/2110.15871/) (2021)
- [MedReadCtrl: Personalizing medical text generation with readability-controlled instruction learning](https://mlsanity.com/paper/2507.07419/) (2025)
- [Exploring Cross-lingual Textual Style Transfer with Large Multilingual Language Models](https://mlsanity.com/paper/2206.02252/) (2022)
- [In What Languages are Generative Language Models the Most Formal? Analyzing Formality Distribution across Languages](https://mlsanity.com/paper/2302.12299/) (2023)
- [UMass-BioNLP at MEDIQA-M3G 2024: DermPrompt -- A Systematic Exploration of Prompt Engineering with GPT-4V for Dermatological Diagnosis](https://mlsanity.com/paper/2404.17749/) (2024)

## Abstract (arXiv)

Models pre-trained on large-scale regular text corpora often do not work well for user-generated data where the language styles differ significantly from the mainstream text. Here we present Context-Aware Rule Injection (CARI), an innovative method for formality style transfer (FST). CARI injects multiple rules into an end-to-end BERT-based encoder and decoder model. It learns to select optimal rules based on context. The intrinsic evaluation showed that CARI achieved the new highest performance on the FST benchmark dataset. Our extrinsic evaluation showed that CARI can greatly improve the regular pre-trained models' performance on several tweet sentiment analysis tasks.