mlsanityData to 7 Oct 2026

LingGen: Scalable Multi-Attribute Linguistic Control via Power-Law Masking

arXiv 2410.24201 ↗First posted 31 Oct 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We present LingGen, a controlled text generation model that allows fine-grained control over a large number of real-valued linguistic attributes. It encodes target attribute values with a dedicated linguistic attribute encoder and conditions the language model by injecting the resulting representation into the language model using the beginning-of-sequence (BOS) embeddings. To improve robustness when controlling different attribute subsets, we introduce P-MASKING, which samples per-example attribute masking rates from a truncated Pareto distribution during training. Across 1-40 control attributes, LingGen achieves the lowest average control error among evaluated methods, while remaining efficient at inference and receiving the highest fluency scores in human evaluation. Ablations show that Pareto-sampled masking and BOS-based injection are effective choices compared to alternative masking and integration variants.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

29 of 49 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  11. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  12. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
17 more
  1. 2017Toward Controlled Generation of Textcited by 271
  2. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 246
  3. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  4. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  5. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  6. 2022Mix and Match: Learning-free Controllable Text Generation using Energy Language Modelscited by 66
  7. 2020Exploring Controllable Text Generation Techniquescited by 55
  8. 2020PMI-Masking: Principled masking of correlated spanscited by 54
  9. 2022Quality Controlled Paraphrase Generationcited by 31
  10. 2023PREADD: Prefix-Adaptive Decoding for Controlled Text Generationcited by 29
  11. 2022Composable Text Controls in Latent Space with ODEscited by 28
  12. 2023BOLT: Fast Energy-based Controlled Text Generation with Tunable Biasescited by 24
  13. 2024Personalized Text Generation with Fine-Grained Linguistic Controlcited by 20
  14. 2023LFTK: Handcrafted Features in Computational Linguisticscited by 19
  15. 2023MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Spacecited by 12
  16. 2022An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generationcited by 10
  17. 2024LiFi: Lightweight Controlled Text Generation with Fine-Grained Control Codescited by 2

Built on by

Not cited yet.