# STEER: Unified Style Transfer with Expert Reinforcement

arXiv: https://arxiv.org/abs/2311.07167 · HTML: https://mlsanity.com/paper/2311.07167/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 59 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 30 of 46)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://mlsanity.com/paper/2212.10465/) (2022)
- [Style Transfer Through Back-Translation](https://mlsanity.com/paper/1804.09000/) (2018)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://mlsanity.com/paper/2109.03910/) (2021)
- [Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation](https://mlsanity.com/paper/1905.05621/) (2019)
- [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://mlsanity.com/paper/1909.06694/) (2019)
- [Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer](https://mlsanity.com/paper/1908.09368/) (2019)

## Built on by (most cited first, 14 of 14)

- [Aligning Language Models with Demonstrated Feedback](https://mlsanity.com/paper/2406.00888/) (2024)
- [StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elements](https://mlsanity.com/paper/2408.15666/) (2024)
- [Distilling Text Style Transfer With Self-Explanation From LLMs](https://mlsanity.com/paper/2403.01106/) (2024)
- [Authorship Style Transfer with Policy Optimization](https://mlsanity.com/paper/2403.08043/) (2024)
- [DetoxLLM: A Framework for Detoxification with Explanations](https://mlsanity.com/paper/2402.15951/) (2024)
- [Keep It Private: Unsupervised Privatization of Online Text](https://mlsanity.com/paper/2405.10260/) (2024)
- [Learning to Generate Text in Arbitrary Writing Styles](https://mlsanity.com/paper/2312.17242/) (2023)
- [Style Transfer with Multi-iteration Preference Optimization](https://mlsanity.com/paper/2406.11581/) (2024)
- [Tuning-Free Personalized Alignment via Trial-Error-Explain In-Context Learning](https://mlsanity.com/paper/2502.08972/) (2025)
- [Automatic Input Rewriting Improves Translation with Large Language Models](https://mlsanity.com/paper/2502.16682/) (2025)
- [Test-Time Steering for Lossless Text Compression via Weighted Product of Experts](https://mlsanity.com/paper/2511.10660/) (2025)
- [Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metrics](https://mlsanity.com/paper/2502.15022/) (2025)
- [Policy Learning with a Natural Language Action Space: A Causal Approach](https://mlsanity.com/paper/2502.17538/) (2025)
- [HyperStyler: Low-resource Authorship Style Transfer via Context-aware Style Navigation and Hypernetworks](https://mlsanity.com/paper/2609.02772/) (2026)

## Abstract (arXiv)

While text style transfer has many applications across natural language processing, the core premise of transferring from a single source style is unrealistic in a real-world setting. In this work, we focus on arbitrary style transfer: rewriting a text from an arbitrary, unknown style to a target style. We propose STEER: Unified Style Transfer with Expert Reinforcement, a unified frame-work developed to overcome the challenge of limited parallel data for style transfer. STEER involves automatically generating a corpus of style-transfer pairs using a product of experts during decoding. The generated offline data is then used to pre-train an initial policy before switching to online, off-policy reinforcement learning for further improvements via fine-grained reward signals. STEER is unified and can transfer to multiple target styles from an arbitrary, unknown source style, making it particularly flexible and efficient. Experimental results on a challenging dataset with text from a diverse set of styles demonstrate state-of-the-art results compared to competitive baselines. Remarkably, STEER outperforms the 175B parameter instruction-tuned GPT-3 on overall style transfer quality, despite being 226 times smaller in size. We also show STEER is robust, maintaining its style transfer capabilities on out-of-domain data, and surpassing nearly all baselines across various styles. The success of our method highlights the potential of RL algorithms when augmented with controllable decoding to overcome the challenge of limited data supervision.