# Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewriting

arXiv: https://arxiv.org/abs/2305.14755 · HTML: https://mlsanity.com/paper/2305.14755/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 94 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 30 of 59)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [HateBERT: Retraining BERT for Abusive Language Detection in English](https://mlsanity.com/paper/2010.12472/) (2020)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Deep Learning for Text Style Transfer: A Survey](https://mlsanity.com/paper/2011.00416/) (2020)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [A Knowledge-Grounded Neural Conversation Model](https://mlsanity.com/paper/1702.01932/) (2017)
- [Dialogue Natural Language Inference](https://mlsanity.com/paper/1811.00671/) (2018)
- [Learning to Write with Cooperative Discriminators](https://mlsanity.com/paper/1805.06087/) (2018)
- [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://mlsanity.com/paper/2104.12250/) (2021)
- [ProsocialDialog: A Prosocial Backbone for Conversational Agents](https://mlsanity.com/paper/2205.12688/) (2022)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://mlsanity.com/paper/2109.03910/) (2021)
- [The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://mlsanity.com/paper/2109.06835/) (2021)
- [Do Massively Pretrained Language Models Make Better Storytellers?](https://mlsanity.com/paper/1909.10705/) (2019)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)

## Built on by (most cited first, 6 of 6)

- [Words Like Knives: Backstory-Personalized Modeling and Detection of Violent Communication](https://mlsanity.com/paper/2505.21451/) (2025)
- [Promoting Constructive Deliberation: Reframing for Receptiveness](https://mlsanity.com/paper/2405.15067/) (2024)
- [Dr Genre: Reinforcement Learning from Decoupled LLM Feedback for Generic Text Rewriting](https://mlsanity.com/paper/2503.06781/) (2025)
- [Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Non-Literal Intent Resolution in LLMs](https://mlsanity.com/paper/2405.08760/) (2024)
- [A Survey of Text Style Transfer: Applications and Ethical Implications](https://mlsanity.com/paper/2407.16737/) (2024)
- [Detoxify: A framework for abusive text transformation using LLMs](https://mlsanity.com/paper/2507.10177/) (2025)

## Abstract (arXiv)

Most existing stylistic text rewriting methods and evaluation metrics operate on a sentence level, but ignoring the broader context of the text can lead to preferring generic, ambiguous, and incoherent rewrites. In this paper, we investigate integrating the preceding textual context into both the textitrewriting and textitevaluation stages of stylistic text rewriting, and introduce a new composite contextual evaluation metric textttCtxSimFit that combines similarity to the original sentence with contextual cohesiveness. We comparatively evaluate non-contextual and contextual rewrites in formality, toxicity, and sentiment transfer tasks. Our experiments show that humans significantly prefer contextual rewrites as more fitting and natural over non-contextual ones, yet existing sentence-level automatic metrics (e.g., ROUGE, SBERT) correlate poorly with human preferences (ρ=0--0.3). In contrast, human preferences are much better reflected by both our novel textttCtxSimFit (ρ=0.7--0.9) as well as proposed context-infused versions of common metrics (ρ=0.4--0.7). Overall, our findings highlight the importance of integrating context into the generation and especially the evaluation stages of stylistic text rewriting.