mlsanityData to 14 Oct 2026

Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer

arXiv 2105.06947 ↗First posted 14 May 2021cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Scarcity of parallel data causes formality style transfer models to have scarce success in preserving content. We show that fine-tuning pre-trained language (GPT-2) and sequence-to-sequence (BART) models boosts content preservation, and that this is possible even with limited amounts of parallel data. Augmenting these models with rewards that target style and content -- the two core aspects of the task -- we achieve a new state-of-the-art.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 14 May 2021.

Builds on

22 of 31 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  5. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  6. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  7. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 248
  8. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 211
  9. 2017Style Transfer in Text: Exploration and Evaluationcited by 146
  10. 2018Disentangled Representation Learning for Non-Parallel Text Style Transfercited by 127
  11. 2018Style Transfer Through Back-Translationcited by 124
  12. 2020A Probabilistic Formulation of Unsupervised Text Style Transfercited by 70
10 more
  1. 2019A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfercited by 65
  2. 2020Parallel Data Augmentation for Formality Style Transfercited by 39
  3. 2019Reinforcement Learning Based Text Style Transfer without Parallel Training Corpuscited by 38
  4. 2016Domain Control for Neural Machine Translationcited by 36
  5. 2018Multi-Task Neural Models for Translating Between Styles Within and Across Languagescited by 28
  6. 2019Semi-supervised Text Style Transfer: Cross Projection in Latent Spacecited by 21
  7. 2019A Hierarchical Reinforced Sequence Operation Method for Unsupervised Text Style Transfercited by 20
  8. 2018Style Transfer as Unsupervised Machine Translationcited by 19
  9. 2020Semi-supervised Formality Style Transfer using Language Model Discriminator and Mutual Information Maximizationcited by 19
  10. 2020Reinforced Rewards Framework for Text Style Transfercited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 11; 2023: 8; 2024: 4; 2025: 1; 2026: 1.

28 papers.

Sort:

By area:

  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  2. 2022End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Modelscited by 107
  3. 2022MVP: Multi-task Supervised Pre-training for Natural Language Generationcited by 16
  4. 2023Prompt-Based Editing for Text Style Transfercited by 16
  5. 2023Deliberate then Generate: Enhanced Prompting Framework for Text Generationcited by 13
  6. 2022Semi-Supervised Formality Style Transfer with Consistency Trainingcited by 12
  7. 2022Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfercited by 12
  8. 2023Multidimensional Evaluation for Text Style Transfer Using ChatGPTcited by 12
  9. 2024Distilling Text Style Transfer With Self-Explanation From LLMscited by 12
  10. 2021Generic resources are what you need: Style transfer tasks without task-specific parallel training datacited by 10
  11. 2022Multilingual Pre-training with Language and Task Adaptation for Multilingual Text Style Transfercited by 8
  12. 2024LLM-based Rewriting of Inappropriate Argumentation using Reinforcement Learning from Machine Feedbackcited by 7
16 more
  1. 2022Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guidingcited by 6
  2. 2023Learning Evaluation Models from Large Language Models for Sequence Generationcited by 6
  3. 2021From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Surveycited by 5
  4. 2023Reducing Sequence Length by Predicting Edit Operations with Large Language Modelscited by 5
  5. 2023VQA-GEN: A Visual Question Answering Benchmark for Domain Generalizationcited by 5
  6. 2022StyleFlow: Disentangle Latent Representations via Normalizing Flow for Unsupervised Text Style Transfercited by 4
  7. 2025SynthDetoxM: Modern LLMs are Few-Shot Parallel Detoxification Data Annotatorscited by 4
  8. 2022Replacing Language Model for Style Transfercited by 3
  9. 2022TextBox 2.0: A Text Generation Library with Pre-trained Language Modelscited by 3
  10. 2022Studying the role of named entities for content preservation in text style transfercited by 1
  11. 2024Positive Text Reframing under Multi-strategy Optimizationcited by 1
  12. 2024Multi-perspective Alignment for Increasing Naturalness in Neural Machine Translationcited by 1
  13. 2022Learning from Bootstrapping and Stepwise Reinforcement Reward: A Semi-Supervised Framework for Text Style Transfercited by 0
  14. 2023HeroNet: A Hybrid Retrieval-Generation Network for Conversational Botscited by 0
  15. 2023PSST: A Benchmark for Evaluation-driven Text Public-Speaking Style Transfercited by 0
  16. 2026RT-SFT: Text Style Transfer from Non-Parallel Corpora by Roundtrip Translationcited by 0