# DIRECTOR: Generator-Classifiers For Supervised Language Modeling

arXiv: https://arxiv.org/abs/2206.07694 · HTML: https://mlsanity.com/paper/2206.07694/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [BASE Layers: Simplifying Training of Large, Sparse Models](https://mlsanity.com/paper/2103.16716/) (2021)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [Challenges in Detoxifying Language Models](https://mlsanity.com/paper/2109.07445/) (2021)
- [Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills](https://mlsanity.com/paper/2004.08449/) (2020)
- [I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling](https://mlsanity.com/paper/2012.13391/) (2020)
- [Controlling Style in Generated Dialogue](https://mlsanity.com/paper/2009.10855/) (2020)
- [CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation](https://mlsanity.com/paper/2204.00862/) (2022)
- [Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity](https://mlsanity.com/paper/2112.05843/) (2021)
- [A Simple Contrastive Learning Objective for Alleviating Neural Text Degeneration](https://mlsanity.com/paper/2205.02517/) (2022)
- [Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONES](https://mlsanity.com/paper/2205.00704/) (2022)

## Built on by (most cited first, 30 of 38)

- [BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://mlsanity.com/paper/2208.03188/) (2022)
- [Controlled Decoding from Language Models](https://mlsanity.com/paper/2310.17022/) (2023)
- [Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss](https://mlsanity.com/paper/2312.16682/) (2023)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://mlsanity.com/paper/2305.00955/) (2023)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [Critic-Guided Decoding for Controlled Text Generation](https://mlsanity.com/paper/2212.10938/) (2022)
- [KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detection](https://mlsanity.com/paper/2310.09044/) (2023)
- [Using In-Context Learning to Improve Dialogue Safety](https://mlsanity.com/paper/2302.00871/) (2023)
- [Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning](https://mlsanity.com/paper/2306.03350/) (2023)
- [The CRINGE Loss: Learning what language not to model](https://mlsanity.com/paper/2211.05826/) (2022)
- [Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback](https://mlsanity.com/paper/2208.03270/) (2022)
- [DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generation](https://mlsanity.com/paper/2210.09551/) (2022)
- [When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels](https://mlsanity.com/paper/2210.15893/) (2022)
- [Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast](https://mlsanity.com/paper/2405.14507/) (2024)
- [Controlled Text Generation for Large Language Model with Dynamic Attribute Graphs](https://mlsanity.com/paper/2402.11218/) (2024)
- [On Almost Surely Safe Alignment of Large Language Models at Inference-Time](https://mlsanity.com/paper/2502.01208/) (2025)
- [RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework](https://mlsanity.com/paper/2410.19109/) (2024)
- [Improving Open Language Models by Learning from Organic Interactions](https://mlsanity.com/paper/2306.04707/) (2023)
- [KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning](https://mlsanity.com/paper/2211.16773/) (2022)
- [Decoupling Task-Solving and Output Formatting in LLM Generation](https://mlsanity.com/paper/2510.03595/) (2025)
- [Learn What NOT to Learn: Towards Generative Safety in Chatbots](https://mlsanity.com/paper/2304.11220/) (2023)
- [Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond](https://mlsanity.com/paper/2305.12289/) (2023)
- [DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines](https://mlsanity.com/paper/2212.10557/) (2022)
- [System-Level Natural Language Feedback](https://mlsanity.com/paper/2306.13588/) (2023)
- [LiFi: Lightweight Controlled Text Generation with Fine-Grained Control Codes](https://mlsanity.com/paper/2402.06930/) (2024)
- [ECO Decoding: Entropy-Based Control for Controllability and Fluency in Controllable Dialogue Generation](https://mlsanity.com/paper/2511.01568/) (2025)
- [Controlling Reading Ease with Gaze-Guided Text Generation](https://mlsanity.com/paper/2601.17781/) (2026)
- [Semantic Space Grounded Weighted Decoding for Multi-Attribute Controllable Dialogue Generation](https://mlsanity.com/paper/2305.02820/) (2023)
- [Successor Features for Efficient Multisubject Controlled Text Generation](https://mlsanity.com/paper/2311.04921/) (2023)

## Abstract (arXiv)

Current language models achieve low perplexity but their resulting generations still suffer from toxic responses, repetitiveness and contradictions. The standard language modeling setup fails to address these issues. In this paper, we introduce a new architecture, sc Director, that consists of a unified generator-classifier with both a language modeling and a classification head for each output token. Training is conducted jointly using both standard language modeling data, and data labeled with desirable and undesirable sequences. Experiments in several settings show that the model has competitive training and decoding speed compared to standard language models while yielding superior results, alleviating known issues while maintaining generation quality. It also outperforms existing model guiding approaches in terms of both accuracy and efficiency.