# On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility

arXiv: https://arxiv.org/abs/2605.01357 · HTML: https://mlsanity.com/paper/2605.01357/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 2 May 2026.

## Builds on (most cited first, 22 of 22)

- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs](https://mlsanity.com/paper/2408.07055/) (2024)
- [Correcting Length Bias in Neural Machine Translation](https://mlsanity.com/paper/1808.10006/) (2018)
- [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://mlsanity.com/paper/1908.10090/) (2019)
- [When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs](https://mlsanity.com/paper/2505.11423/) (2025)
- [The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input](https://mlsanity.com/paper/2501.03200/) (2025)
- [AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentation](https://mlsanity.com/paper/2403.02959/) (2024)
- [With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation](https://mlsanity.com/paper/2401.11504/) (2024)
- [Suri: Multi-constraint Instruction Following for Long-form Text Generation](https://mlsanity.com/paper/2406.19371/) (2024)
- [LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation](https://mlsanity.com/paper/2501.05414/) (2025)
- [Language Models can Self-Lengthen to Generate Long Texts](https://mlsanity.com/paper/2410.23933/) (2024)
- [LLM×MapReduce: Simplified Long-Sequence Processing using Large Language Models](https://mlsanity.com/paper/2410.09342/) (2024)
- [MARS-SQL: A multi-agent reinforcement learning framework for Text-to-SQL](https://mlsanity.com/paper/2511.01008/) (2025)
- [LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning](https://mlsanity.com/paper/2506.18841/) (2025)
- [LIFEBench: Evaluating Length Instruction Following in Large Language Models](https://mlsanity.com/paper/2505.16234/) (2025)
- [ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialects](https://mlsanity.com/paper/2505.17231/) (2025)
- [Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization](https://mlsanity.com/paper/2502.14496/) (2025)
- [LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information](https://mlsanity.com/paper/2502.02095/) (2025)
- [ClinTutor-R1: Advancing Scalable and Robust One-to-Many Alignment in Clinical Socratic Education](https://mlsanity.com/paper/2512.05671/) (2025)
- [LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models](https://mlsanity.com/paper/2502.14834/) (2025)
- [Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation](https://mlsanity.com/paper/2109.06253/) (2021)

## Built on by (most cited first, 4 of 4)

- [An Analysis of Training-Free Self-Reported Confidence in Language Models](https://mlsanity.com/paper/2609.20541/) (2026)
- [Mind or Message? Auditing Theory of Mind in Multi-Agent Social Simulation](https://mlsanity.com/paper/2609.24146/) (2026)
- [Scaling Long-Form Story Generation via Narrative State Tracking](https://mlsanity.com/paper/2609.35759/) (2026)
- [EvoQuantLab: Training-Free Dual-Loop Evolution for Auditable Quantitative Research](https://mlsanity.com/paper/2610.17130/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) excel at long-context understanding but exhibit significant limitations in long-form generation. Existing studies primarily focus on single-generation quality, generally overlooking the volatility of the output. This volatility not only leads to significant computational costs but also severely impacts the models' reliable application. To address this gap, our work unfolds in three stages: benchmarking, probing, and mitigation. We first propose the VOlatility in Long-form Text Benchmark (VOLTBench), a novel heterogeneous-task benchmark designed to systematically quantify the length volatility of long-form generation. Subsequently, by analyzing attention traces, we conduct an in-depth probe to identify several common internal patterns that cause this volatility. Finally, to mitigate long-form output volatility, we propose Stable Generation via Logits Boosting (GLoBo), a lightweight decoding-stage optimization strategy, designed to significantly enhance both the length accuracy and stability of long-form generation without additional training. Extensive experiments on VOLTBench provide the first systematic confirmation of severe long-form output instability in mainstream models and validate that our proposed method successfully improves the mean output length of the base model by 148% and reduces the length volatility by 69%, while maintaining high generation quality.