mlsanityData to 7 Oct 2026

Quark: Controllable Text Generation with Reinforced Unlearning

arXiv 2205.13636 ↗First posted 26 May 2022cs.CL cs.LGcited by 236

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Large-scale language models often learn behaviors that are misaligned with user expectations. Generated text may contain offensive or toxic language, contain significant repetition, or be of a different sentiment than desired by the user. We consider the task of unlearning these misalignments by fine-tuning the language model on signals of what not to do. We introduce Quantized Reward Konditioning (Quark), an algorithm for optimizing a reward function that quantifies an (un)wanted property, while not straying too far from the original model. Quark alternates between (i) collecting samples with the current language model, (ii) sorting them into quantiles based on reward, with each quantile identified by a reward token prepended to the language model's input, and (iii) using a standard language modeling loss on samples from each quantile conditioned on its reward token, while remaining nearby the original language model via a KL-divergence penalty. By conditioning on a high-reward token at generation time, the model generates text that exhibits less of the unwanted property. For unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both strong baselines and state-of-the-art reinforcement learning methods like PPO (Schulman et al. 2017), while relying only on standard language modeling primitives.

Key facts. Cited by 236 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 76 of its 100 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

76 of 100 references.

References · 76

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
38 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2018Hierarchical Neural Story Generationcited by 1,182
  9. 2018Model Cards for Model Reportingcited by 1,068
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  13. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  14. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  15. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  16. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  17. 2016Self-critical Sequence Training for Image Captioningcited by 544
  18. 2021Fast Model Editing at Scalecited by 491
  19. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  20. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  21. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  22. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  23. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  24. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  25. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  26. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  27. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  28. 2019Vision-and-Dialog Navigationcited by 246
  29. 2022A Contrastive Framework for Neural Text Generationcited by 224
  30. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  31. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  32. 2022Online Decision Transformercited by 194
  33. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  34. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  35. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  36. 2020A Distributional Approach to Controlled Text Generationcited by 117
  37. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  38. 2020Towards Controllable Biases in Language Generationcited by 112

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 73; 2024: 77; 2025: 48; 2026: 23.

236 papers.

Sort:

By area:

  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2022Editing Models with Task Arithmeticcited by 1,076
  4. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  7. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  9. 2022A Survey of Machine Unlearningcited by 298
  10. 2023Large Language Model Unlearningcited by 288
  11. 2022Generating Sequences by Learning to Self-Correctcited by 271
  12. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
38 more
  1. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  2. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  3. 2023Pretraining Language Models with Human Preferencescited by 245
  4. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  5. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  6. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  7. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  8. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  9. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  10. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  11. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  12. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  13. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  14. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  15. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 144
  16. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  17. 2024Direct Preference Optimization with an Offsetcited by 122
  18. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  19. 2022Latent Diffusion for Language Generationcited by 110
  20. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  21. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  22. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  23. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  24. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
  25. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
  26. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  27. 2023A Survey on Fairness in Large Language Modelscited by 91
  28. 2023Machine Unlearning: Solutions and Challengescited by 85
  29. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  30. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  31. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  32. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 77
  33. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  34. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  35. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 59
  36. 2024Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypescited by 58
  37. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  38. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48

50 most cited of 236 All 236 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Critical Look At Tokenwise Reward-Guided Text Generation
  2. 2025Text2Grad: Reinforcement Learning from Natural Language Feedback
  3. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs
  4. 2020Text Generation by Learning from Demonstrations
  5. 2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference
  6. 2026Automatically Finding Reward Model Biases
  7. 2026Characterising Toxicity in Generative Large Language Models
  8. 2026Robust Reward Modeling for Large Language Models via Causal Decomposition