# LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigm

arXiv: https://arxiv.org/abs/2502.19103 · HTML: https://mlsanity.com/paper/2502.19103/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Builds on (most cited first, 19 of 19)

- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs](https://mlsanity.com/paper/2408.07055/) (2024)
- [Make Your LLM Fully Utilize the Context](https://mlsanity.com/paper/2404.16811/) (2024)
- [LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA](https://mlsanity.com/paper/2409.02897/) (2024)
- [A Comparative Study on Reasoning Patterns of OpenAI's o1 Model](https://mlsanity.com/paper/2410.13639/) (2024)
- [LongForm: Effective Instruction Tuning with Reverse Instructions](https://mlsanity.com/paper/2304.08460/) (2023)
- [HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models](https://mlsanity.com/paper/2409.16191/) (2024)
- [Synthetic Dialogue Dataset Generation using LLM Agents](https://mlsanity.com/paper/2401.17461/) (2024)
- [Chinese Open Instruction Generalist: A Preliminary Release](https://mlsanity.com/paper/2304.07987/) (2023)
- [Suri: Multi-constraint Instruction Following for Long-form Text Generation](https://mlsanity.com/paper/2406.19371/) (2024)
- [LongReward: Improving Long-context Large Language Models with AI Feedback](https://mlsanity.com/paper/2410.21252/) (2024)
- [Language Models can Self-Lengthen to Generate Long Texts](https://mlsanity.com/paper/2410.23933/) (2024)
- [Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generation](https://mlsanity.com/paper/2408.13586/) (2024)
- [SkyScript-100M: 1,000,000,000 Pairs of Scripts and Shooting Scripts for Short Drama](https://mlsanity.com/paper/2408.09333/) (2024)
- [Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularity](https://mlsanity.com/paper/2405.16579/) (2024)

## Built on by (most cited first, 14 of 14)

- [A Survey on LLM Mid-Training](https://mlsanity.com/paper/2510.23081/) (2025)
- [FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agents](https://mlsanity.com/paper/2507.16248/) (2025)
- [Agentic Systems in Radiology: Design, Applications, Evaluation, and Challenges](https://mlsanity.com/paper/2510.09404/) (2025)
- [FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality](https://mlsanity.com/paper/2607.12252/) (2026)
- [Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generation](https://mlsanity.com/paper/2604.10741/) (2026)
- [Conversational Process Model Redesign](https://mlsanity.com/paper/2505.05453/) (2025)
- [MemoryRewardBench: Benchmarking Reward Models for Long-Term Memory Management in Large Language Models](https://mlsanity.com/paper/2601.11969/) (2026)
- [From Personas to Plot: Character-Grounded Multi-Agent Story Generation for Long-Form Narratives](https://mlsanity.com/paper/2607.00918/) (2026)
- [INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detection](https://mlsanity.com/paper/2602.18448/) (2026)
- [NARRA-Gym for Evaluating Interactive Narrative Agents](https://mlsanity.com/paper/2605.08503/) (2026)
- [COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for Alignment with Human Values](https://mlsanity.com/paper/2504.05535/) (2025)
- [Measuring Information Distortion in Hierarchical Ultra long Novel Reconstruction:The Optimal Expansion Ratio](https://mlsanity.com/paper/2505.12572/) (2025)
- [SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions](https://mlsanity.com/paper/2605.07102/) (2026)
- [SAGE: A Hierarchical Framework for Evaluating Interpretive Literary Quality in Narratives](https://mlsanity.com/paper/2609.06611/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks, yet their ability to generate long-form content remains poorly understood and evaluated. Our analysis reveals that current LLMs struggle with length requirements and information density in long-text generation, with performance deteriorating as text length increases. To quantitively locate such a performance degradation and provide further insights on model development, we present LongEval, a benchmark that evaluates long-text generation through both direct and plan-based generation paradigms, inspired by cognitive and linguistic writing models. The comprehensive experiments in this work reveal interesting findings such as that while model size correlates with generation ability, the small-scale model (e.g., LongWriter), well-trained on long texts, has comparable performance. All code and datasets are released in https://github.com/Wusiwei0410/LongEval.