# Air-Decoding: Attribute Distribution Reconstruction for Decoding-Time Controllable Text Generation

arXiv: https://arxiv.org/abs/2310.14892 · HTML: https://mlsanity.com/paper/2310.14892/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [Controllable Natural Language Generation with Contrastive Prefixes](https://mlsanity.com/paper/2202.13257/) (2022)
- [Controlled Text Generation as Continuous Optimization with Multiple Constraints](https://mlsanity.com/paper/2108.01850/) (2021)
- [Mix and Match: Learning-free Controllable Text Generation using Energy Language Models](https://mlsanity.com/paper/2203.13299/) (2022)
- [CoCon: A Self-Supervised Approach for Controlled Text Generation](https://mlsanity.com/paper/2006.03535/) (2020)
- [A Distributional Lens for Multi-Aspect Controllable Text Generation](https://mlsanity.com/paper/2210.02889/) (2022)
- [Plug-and-Play Conversational Models](https://mlsanity.com/paper/2010.04344/) (2020)
- [DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generation](https://mlsanity.com/paper/2210.09551/) (2022)
- [CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation](https://mlsanity.com/paper/2204.00862/) (2022)
- [Attribute Alignment: Controlling Text Generation from Pre-trained Language Models](https://mlsanity.com/paper/2103.11070/) (2021)
- [Tailor: A Prompt-Based Approach to Attribute-Based Controlled Text Generation](https://mlsanity.com/paper/2204.13362/) (2022)
- [Plug-and-Blend: A Framework for Controllable Story Generation with Blended Control Codes](https://mlsanity.com/paper/2104.04039/) (2021)
- [ZeroGen: Zero-shot Multimodal Controllable Text Generation with Multiple Oracles](https://mlsanity.com/paper/2306.16649/) (2023)

## Built on by (most cited first, 10 of 10)

- [Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference](https://mlsanity.com/paper/2406.08607/) (2024)
- [Controlled Text Generation for Large Language Model with Dynamic Attribute Graphs](https://mlsanity.com/paper/2402.11218/) (2024)
- [DESTEIN: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion](https://mlsanity.com/paper/2404.10464/) (2024)
- [Controllable Text Generation in the Instruction-Tuning Era](https://mlsanity.com/paper/2405.01490/) (2024)
- [FreeCtrl: Constructing Control Centers with Feedforward Layers for Learning-Free Controllable Text Generation](https://mlsanity.com/paper/2406.09688/) (2024)
- [C3TG: Conflict-aware, Composite, and Collaborative Controlled Text Generation](https://mlsanity.com/paper/2511.09292/) (2025)
- [Don't Let It Fade: Preserving Edits in Diffusion Language Models via Token Timestep Allocation](https://mlsanity.com/paper/2510.26200/) (2025)
- [LaPA2: Length-Aware Prefix and Prompt Attention Augmentation for Long-Form Controllable Text Generation](https://mlsanity.com/paper/2508.04047/) (2025)
- [Towards Lightweight, Adaptive and Attribute-Aware Multi-Aspect Controllable Text Generation with Large Language Models](https://mlsanity.com/paper/2502.13474/) (2025)
- [AgentCTG: Harnessing Multi-Agent Collaboration for Fine-Grained Precise Control in Text Generation](https://mlsanity.com/paper/2509.13677/) (2025)

## Abstract (arXiv)

Controllable text generation (CTG) aims to generate text with desired attributes, and decoding-time-based methods have shown promising performance on this task. However, in this paper, we identify the phenomenon of Attribute Collapse for the first time. It causes the fluency of generated text to rapidly decrease when the control strength exceeds a critical value, rendering the text completely unusable. This limitation hinders the effectiveness of decoding methods in achieving high levels of controllability. To address this problem, we propose a novel lightweight decoding framework named Air-Decoding. Its main idea is reconstructing the attribute distributions to balance the weights between attribute words and non-attribute words to generate more fluent text. Specifically, we train prefixes by prefix-tuning to obtain attribute distributions. Then we design a novel attribute distribution reconstruction method to balance the obtained distributions and use the reconstructed distributions to guide language models for generation, effectively avoiding the issue of Attribute Collapse. Experiments on multiple CTG tasks prove that our method achieves a new state-of-the-art control performance.