mlsanityData to 14 Oct 2026

LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs

arXiv 2408.07055 ↗First posted 13 Aug 2024cs.CL cs.LGcited by 144

Not on the map. Why?

Abstract (arXiv)

Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability. Our code & models are at: https://github.com/THUDM/LongWriter.

Key facts. Cited by 144 papers in the map (data through 14 Oct 2026). Builds on 40 of its 49 references that are in the map. First posted on arXiv 13 Aug 2024.

Builds on

40 of 49 references.

References · 40

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2023Mistral 7Bcited by 3,164
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
28 more
  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  3. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  5. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  6. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  7. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  9. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  11. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  12. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  13. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 235
  14. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  15. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  16. 2024Distilling System 2 into System 1cited by 159
  17. 2023Self-Alignment with Instruction Backtranslationcited by 151
  18. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  19. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  20. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  21. 2024Long Context Compression with Activation Beaconcited by 104
  22. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  23. 2024Following Length Constraints in Instructionscited by 46
  24. 2024Suri: Multi-constraint Instruction Following for Long-form Text Generationcited by 34
  25. 2024Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Modelscited by 27
  26. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  27. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedbackcited by 25
  28. 2024Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Modelscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 71; 2026: 59.

144 papers.

Sort:

By area:

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  2. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  3. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  4. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 106
  5. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  6. 2024Agents' Room: Narrative Generation through Multi-step Collaborationcited by 57
  7. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 52
  8. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  9. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  10. 2025WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Researchcited by 46
  11. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 44
  12. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 42
38 more
  1. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reductioncited by 39
  2. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  3. 2024Denial-of-Service Poisoning Attacks against Large Language Modelscited by 35
  4. 2025Hogwild! Inference: Parallel LLM Generation via Concurrent Attentioncited by 32
  5. 2025Reverse-Engineered Reasoning for Open-Ended Generationcited by 32
  6. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 31
  7. 2024Generating Long-form Story Using Dynamic Hierarchical Outlining with Memory-Enhancementcited by 30
  8. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 29
  9. 2024Language Models can Self-Lengthen to Generate Long Textscited by 24
  10. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  11. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  12. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 20
  13. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMscited by 20
  14. 2025RAPID: Efficient Retrieval-Augmented Long Text Generation with Writing Planning and Information Discoverycited by 19
  15. 2025Reasoning Path Compression: Compressing Generation Trajectories for Efficient LLM Reasoningcited by 17
  16. 2026ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Rankingcited by 17
  17. 2024AutoPatent: A Multi-Agent Framework for Automatic Patent Generationcited by 16
  18. 2025Beyond Outlining: Heterogeneous Recursive Planning for Adaptive Long-form Writing with Language Modelscited by 16
  19. 2025LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigmcited by 14
  20. 2025Shifting Long-Context LLMs Research from Input to Outputcited by 14
  21. 2025Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generationcited by 14
  22. 2025LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learningcited by 13
  23. 2025FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agentscited by 12
  24. 2025LongCaptioning: Unlocking the Power of Long Video Caption Generation in Large Multimodal Modelscited by 11
  25. 2025LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMscited by 11
  26. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  27. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  28. 2025STORYTELLER: An Enhanced Plot-Planning Framework for Coherent and Cohesive Story Generationcited by 10
  29. 2025Writing-RL: Advancing Long-form Writing via Adaptive Curriculum Reinforcement Learningcited by 9
  30. 2025ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learningcited by 9
  31. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
  32. 2025LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Informationcited by 8
  33. 2025NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tablescited by 8
  34. 2025Frankentext: Stitching random text fragments into long-form narrativescited by 8
  35. 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 8
  36. 2025AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelismcited by 7
  37. 2025Understanding DeepResearch via Reportscited by 7
  38. 2026DREAM: Deep Research Evaluation with Agentic Metricscited by 7

50 most cited of 144 All 144 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation
  2. 2025LIFEBench: Evaluating Length Instruction Following in Large Language Models
  3. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration
  4. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMs
  5. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  6. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
  7. 2025Systematic Evaluation of Optimization Techniques for Long-Context Language Models
  8. 2025LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions