mlsanityData to 7 Oct 2026

LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs

arXiv 2409.02076 ↗First posted 3 Sep 2024cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

Current benchmarks like Needle-in-a-Haystack (NIAH), Ruler, and Needlebench focus on models' ability to understand long-context input sequences but fail to capture a critical dimension: the generation of high-quality long-form text. Applications such as design proposals, technical documentation, and creative writing rely on coherent, instruction-following outputs over extended sequences - a challenge that existing benchmarks do not adequately address. To fill this gap, we introduce LongGenBench, a novel benchmark designed to rigorously evaluate large language models' (LLMs) ability to generate long text while adhering to complex instructions. Through tasks requiring specific events or constraints within generated text, LongGenBench evaluates model performance across four distinct scenarios, three instruction types, and two generation-lengths (16K and 32K tokens). Our evaluation of ten state-of-the-art LLMs reveals that, despite strong results on Ruler, all models struggled with long text generation on LongGenBench, particularly as text length increased. This suggests that current LLMs are not yet equipped to meet the demands of real-world, long-form text generation.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

45 of 58 references.

References · 45

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
33 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2018Hierarchical Neural Story Generationcited by 1,182
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  5. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  7. 2019ELI5: Long Form Question Answeringcited by 567
  8. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  9. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  10. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  11. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  12. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  13. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  14. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  15. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  16. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  17. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  18. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  19. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  20. 2019Strategies for Structuring Story Generationcited by 124
  21. 2024Make Your LLM Fully Utilize the Contextcited by 118
  22. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  23. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  24. 2024LongLaMP: A Benchmark for Personalized Long-form Text Generationcited by 91
  25. 2020MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Modelscited by 74
  26. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  27. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  28. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  29. 2020PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generationcited by 35
  30. 2022PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generationcited by 30
  31. 2024PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Modelscited by 23
  32. 2022ChapterBreak: A Challenge Dataset for Long-Range Language Modelscited by 18
  33. 2020Plan ahead: Self-Supervised Text Planning for Paragraph Completion Taskcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 23.

47 papers.

Sort:

By area:

  1. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  2. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  3. 2025WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Researchcited by 44
  4. 2025LongCodeBench: Evaluating Coding LLMs at 1M Context Windowscited by 37
  5. 2025Learning to Reason for Long-Form Story Generationcited by 34
  6. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  7. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  8. 2025SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generationcited by 14
  9. 2025Shifting Long-Context LLMs Research from Input to Outputcited by 13
  10. 2025LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learningcited by 13
  11. 2025Minerva: A Programmable Memory Test Benchmark for Language Modelscited by 9
  12. 2025ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learningcited by 9
35 more
  1. 2025Frankentext: Stitching random text fragments into long-form narrativescited by 8
  2. 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 7
  3. 2026DREAM: Deep Research Evaluation with Agentic Metricscited by 7
  4. 2025Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarizationcited by 6
  5. 2026Lost in Stories: Consistency Bugs in Long Story Generation by LLMscited by 6
  6. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttentioncited by 6
  7. 2025Align to Structure: Aligning Large Language Models with Structural Informationcited by 5
  8. 2025SuperWriter: Reflection-Driven Long-Form Generation with Large Language Modelscited by 5
  9. 2025FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Managementcited by 4
  10. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMscited by 4
  11. 2026Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generationcited by 4
  12. 2026Parallel Context Compaction for Long-Horizon LLM Agent Servingcited by 4
  13. 2025Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigationcited by 3
  14. 2025An evaluation of LLMs for generating movie reviews: GPT-4o, Gemini-2.0 and DeepSeek-V3cited by 2
  15. 2025LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiabilitycited by 2
  16. 2026MemoryRewardBench: Benchmarking Reward Models for Long-Term Memory Management in Large Language Modelscited by 2
  17. 2026ReportLogic: Evaluating Logical Quality in Deep Research Reportscited by 2
  18. 2026DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Modelscited by 2
  19. 2026Long-form RewardBench: Evaluating Reward Models for Long-form Generationcited by 2
  20. 2025Monocle: Hybrid Local-Global In-Context Evaluation for Long-Text Generation with Uncertainty-Based Active Learningcited by 1
  21. 2025Power Law Guided Dynamic Sifting for Efficient Attentioncited by 1
  22. 2026Bridging the Arithmetic Gap: The Cognitive Complexity Benchmark and Financial-PoT for Robust Financial Reasoningcited by 1
  23. 2026IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement Learningcited by 1
  24. 2026MAC-Attention: a Match-Amend-Complete Scheme for Fast and Accurate Attention Computationcited by 1
  25. 2025Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statementscited by 0
  26. 2026UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inferencecited by 0
  27. 2026LLMs Should Incorporate Explicit Mechanisms for Human Empathycited by 0
  28. 2026Unified KV Pooling to Accelerate Long-Context LLM Servingcited by 0
  29. 2026GEIS: A Generation-Evaluation-Improvement Loop of Agent Skills for Long-Form Article Generationcited by 0
  30. 2026FinReportBench: Measuring and Improving Institution-Grade Financial Report Generationcited by 0
  31. 2026LogicTree-RAG: Logic Tree-guided Retrieval-Augmented Generation for Long-form Patent Draftingcited by 0
  32. 2026Scaling Long-Form Story Generation via Narrative State Trackingcited by 0
  33. 2026Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptationscited by 0
  34. 2026SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scalingcited by 0
  35. 2026OVAL: Output-Aware Local Page Bases for KV Cache Retrievalcited by 0