# LaPA2: Length-Aware Prefix and Prompt Attention Augmentation for Long-Form Controllable Text Generation

arXiv: https://arxiv.org/abs/2508.04047 · HTML: https://mlsanity.com/paper/2508.04047/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 30 of 32)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [Token-level Direct Preference Optimization](https://mlsanity.com/paper/2404.11999/) (2024)
- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [A Distributional Approach to Controlled Text Generation](https://mlsanity.com/paper/2012.11635/) (2020)
- [Controllable Natural Language Generation with Contrastive Prefixes](https://mlsanity.com/paper/2202.13257/) (2022)
- [Controllable Text Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2408.12599/) (2024)
- [Controlled Text Generation via Language Model Arithmetic](https://mlsanity.com/paper/2311.14479/) (2023)
- [CoCon: A Self-Supervised Approach for Controlled Text Generation](https://mlsanity.com/paper/2006.03535/) (2020)
- [Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding](https://mlsanity.com/paper/2505.16652/) (2025)
- [DIRECTOR: Generator-Classifiers For Supervised Language Modeling](https://mlsanity.com/paper/2206.07694/) (2022)
- [ROSE Doesn't Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding](https://mlsanity.com/paper/2402.11889/) (2024)
- [DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generation](https://mlsanity.com/paper/2210.09551/) (2022)
- [PREADD: Prefix-Adaptive Decoding for Controlled Text Generation](https://mlsanity.com/paper/2307.03214/) (2023)
- [Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning](https://mlsanity.com/paper/2211.15076/) (2022)
- [Controlled Text Generation for Large Language Model with Dynamic Attribute Graphs](https://mlsanity.com/paper/2402.11218/) (2024)
- [Reinforcement Learning with Token-level Feedback for Controllable Text Generation](https://mlsanity.com/paper/2403.11558/) (2024)
- [RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework](https://mlsanity.com/paper/2410.19109/) (2024)
- [Air-Decoding: Attribute Distribution Reconstruction for Decoding-Time Controllable Text Generation](https://mlsanity.com/paper/2310.14892/) (2023)
- [FreeCtrl: Constructing Control Centers with Feedforward Layers for Learning-Free Controllable Text Generation](https://mlsanity.com/paper/2406.09688/) (2024)
- [IKOD: Mitigating Visual Attention Degradation in Large Vision-Language Models](https://mlsanity.com/paper/2508.03469/) (2025)

## Built on by (most cited first, 1 of 1)

- [OmniDocLayout: Towards Diverse Document Layout Generation via Coarse-to-Fine LLM Learning](https://mlsanity.com/paper/2510.26213/) (2025)

## Abstract (arXiv)

Prefix-based methods have emerged as a promising paradigm for Controllable Text Generation (CTG) due to their parameter efficiency. However, while effective in short sequences, their controllability tends to diminish as the generated sequence grows. In this paper, we identify Attention Dilution as a key factor behind this phenomenon: as the sequence length increases, the attention allocated to the control signal naturally decays due to the softmax mechanism, leading to a "fading" control effect. To address this, we propose LaPA2 (Length-aware Prefix and Prompt Attention Augmentation), a training-free and model-agnostic framework designed to sustain robust control in long contexts. Specifically, LaPA2 employs Length-Aware Logarithmic Scaling to dynamically amplify prefix attention weights, mathematically counteracting the dilution effect, while an optional Contextual Anchor Reinforcement applies synchronized augmentation to prompt tokens, preserving semantic coherence when strong attribute control risks overshadowing the original prompt. LaPA2 is versatile, supporting both soft prefixes (continuous embeddings) and hard prefixes (discrete instructions). Experiments on multiple CTG tasks demonstrate that LaPA2 consistently improves the performance of various prefix-based methods in long-form settings, leading to superior attribute controllability while preserving content relevance and fluency. Our code and data are publicly available at https://github.com/jiabingyang01/LaPA2.