# A Distributional Lens for Multi-Aspect Controllable Text Generation

arXiv: https://arxiv.org/abs/2210.02889 · HTML: https://mlsanity.com/paper/2210.02889/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics](https://mlsanity.com/paper/2202.11705/) (2022)
- [Controlling Linguistic Style Aspects in Neural Language Generation](https://mlsanity.com/paper/1707.02633/) (2017)
- [A Distributional Approach to Controlled Text Generation](https://mlsanity.com/paper/2012.11635/) (2020)
- [Controllable Natural Language Generation with Contrastive Prefixes](https://mlsanity.com/paper/2202.13257/) (2022)
- [Controlled Text Generation as Continuous Optimization with Multiple Constraints](https://mlsanity.com/paper/2108.01850/) (2021)
- [Mix and Match: Learning-free Controllable Text Generation using Energy Language Models](https://mlsanity.com/paper/2203.13299/) (2022)
- [Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders](https://mlsanity.com/paper/1911.03882/) (2019)
- [Attribute Alignment: Controlling Text Generation from Pre-trained Language Models](https://mlsanity.com/paper/2103.11070/) (2021)
- [Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models](https://mlsanity.com/paper/2209.06970/) (2022)
- [Plug and Play Autoencoders for Conditional Text Generation](https://mlsanity.com/paper/2010.02983/) (2020)
- [Tailor: A Prompt-Based Approach to Attribute-Based Controlled Text Generation](https://mlsanity.com/paper/2204.13362/) (2022)

## Built on by (most cited first, 30 of 33)

- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer](https://mlsanity.com/paper/2306.07799/) (2023)
- [Critic-Guided Decoding for Controlled Text Generation](https://mlsanity.com/paper/2212.10938/) (2022)
- [Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents](https://mlsanity.com/paper/2402.00798/) (2024)
- [Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications](https://mlsanity.com/paper/2311.05876/) (2023)
- [Controllable Text Generation via Probability Density Estimation in the Latent Space](https://mlsanity.com/paper/2212.08307/) (2022)
- [Benchmarking Large Language Models on Controllable Generation under Diversified Instructions](https://mlsanity.com/paper/2401.00690/) (2024)
- [Mixture of Soft Prompts for Controllable Data Generation](https://mlsanity.com/paper/2303.01580/) (2023)
- [Reinforcement Learning with Token-level Feedback for Controllable Text Generation](https://mlsanity.com/paper/2403.11558/) (2024)
- [MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Space](https://mlsanity.com/paper/2305.12785/) (2023)
- [An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generation](https://mlsanity.com/paper/2212.09387/) (2022)
- [Air-Decoding: Attribute Distribution Reconstruction for Decoding-Time Controllable Text Generation](https://mlsanity.com/paper/2310.14892/) (2023)
- [Focused Prefix Tuning for Controllable Text Generation](https://mlsanity.com/paper/2306.00369/) (2023)
- [Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networks](https://mlsanity.com/paper/2305.01713/) (2023)
- [FreeCtrl: Constructing Control Centers with Feedforward Layers for Learning-Free Controllable Text Generation](https://mlsanity.com/paper/2406.09688/) (2024)
- [Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncoder](https://mlsanity.com/paper/2210.06230/) (2022)
- [A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning](https://mlsanity.com/paper/2406.14164/) (2024)
- [LiFi: Lightweight Controlled Text Generation with Fine-Grained Control Codes](https://mlsanity.com/paper/2402.06930/) (2024)
- [Multi-Aspect Controllable Text Generation with Disentangled Counterfactual Augmentation](https://mlsanity.com/paper/2405.19958/) (2024)
- [LangVAE and LangSpace: Building and Probing for Language Model VAEs](https://mlsanity.com/paper/2505.00004/) (2025)
- [Semantic Space Grounded Weighted Decoding for Multi-Attribute Controllable Dialogue Generation](https://mlsanity.com/paper/2305.02820/) (2023)
- [ProSwitch: Knowledge-Guided Instruction Tuning to Switch Between Professional and Non-Professional Responses](https://mlsanity.com/paper/2403.09131/) (2024)
- [Controllable Topic-Focused Abstractive Summarization](https://mlsanity.com/paper/2311.06724/) (2023)
- [A Lightweight Multi Aspect Controlled Text Generation Solution For Large Language Models](https://mlsanity.com/paper/2410.14144/) (2024)
- [UltraGen: Extremely Fine-grained Controllable Generation via Attribute Reconstruction and Global Preference Optimization](https://mlsanity.com/paper/2502.12375/) (2025)
- [Funny or Persuasive, but Not Both: Evaluating Fine-Grained Multi-Concept Control in LLMs](https://mlsanity.com/paper/2601.18483/) (2026)
- [Reproducing the Metric-Based Evaluation of a Set of Controllable Text Generation Techniques](https://mlsanity.com/paper/2405.07875/) (2024)
- [Towards Lightweight, Adaptive and Attribute-Aware Multi-Aspect Controllable Text Generation with Large Language Models](https://mlsanity.com/paper/2502.13474/) (2025)
- [Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder](https://mlsanity.com/paper/2506.20083/) (2025)
- [PG-CE: A Progressive Generation Dataset with Constraint Enhancement for Controllable Text Generation](https://mlsanity.com/paper/2509.17669/) (2025)

## Abstract (arXiv)

Multi-aspect controllable text generation is a more challenging and practical task than single-aspect control. Existing methods achieve complex multi-aspect control by fusing multiple controllers learned from single-aspect, but suffer from attribute degeneration caused by the mutual interference of these controllers. To address this, we provide observations on attribute fusion from a distributional perspective and propose to directly search for the intersection areas of multiple attribute distributions as their combination for generation. Our method first estimates the attribute space with an autoencoder structure. Afterward, we iteratively approach the intersections by jointly minimizing distances to points representing different attributes. Finally, we map them to attribute-relevant sentences with a prefix-tuning-based decoder. Experiments on the three-aspect control task, including sentiment, topic, and detoxification aspects, reveal that our method outperforms several strong baselines on attribute relevance and text quality and achieves the SOTA. Further analysis also supplies some explanatory support for the effectiveness of our approach.