# Specializing Small Language Models towards Complex Style Transfer via Latent Attribute Pre-Training

arXiv: https://arxiv.org/abs/2309.10929 · HTML: https://mlsanity.com/paper/2309.10929/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 19 Sep 2023.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Intriguing Properties of Contrastive Losses](https://mlsanity.com/paper/2011.02803/) (2020)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://mlsanity.com/paper/2109.03910/) (2021)
- [Automatically Neutralizing Subjective Bias in Text](https://mlsanity.com/paper/1911.09709/) (2019)
- [Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer](https://mlsanity.com/paper/1908.09368/) (2019)
- [Shakespearizing Modern Language Using Copy-Enriched Sequence-to-Sequence Models](https://mlsanity.com/paper/1707.01161/) (2017)
- [Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach](https://mlsanity.com/paper/1805.05181/) (2018)
- [Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer](https://mlsanity.com/paper/1805.07685/) (2018)
- [Hooks in the Headline: Learning to Generate Headlines with Controlled Styles](https://mlsanity.com/paper/2004.01980/) (2020)
- [TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling](https://mlsanity.com/paper/2010.03802/) (2020)
- [Adapting Language Models for Non-Parallel Author-Stylized Rewriting](https://mlsanity.com/paper/1909.09962/) (2019)

## Built on by (most cited first, 2 of 2)

- [ProSwitch: Knowledge-Guided Instruction Tuning to Switch Between Professional and Non-Professional Responses](https://mlsanity.com/paper/2403.09131/) (2024)
- [SETTP: Style Extraction and Tunable Inference via Dual-level Transferable Prompt Learning](https://mlsanity.com/paper/2407.15556/) (2024)

## Abstract (arXiv)

In this work, we introduce the concept of complex text style transfer tasks, and constructed complex text datasets based on two widely applicable scenarios. Our dataset is the first large-scale data set of its kind, with 700 rephrased sentences and 1,000 sentences from the game Genshin Impact. While large language models (LLM) have shown promise in complex text style transfer, they have drawbacks such as data privacy concerns, network instability, and high deployment costs. To address these issues, we explore the effectiveness of small models (less than T5-3B) with implicit style pre-training through contrastive learning. We also propose a method for automated evaluation of text generation quality based on alignment with human evaluations using ChatGPT. Finally, we compare our approach with existing methods and show that our model achieves state-of-art performances of few-shot text style transfer models.