mlsanityData to 14 Oct 2026

LongAlign: A Recipe for Long Context Alignment of Large Language Models

arXiv 2401.18058 ↗First posted 31 Jan 2024cs.CL cs.LGcited by 106

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Extending large language models to effectively handle long contexts requires instruction fine-tuning on input sequences of similar length. To address this, we present LongAlign -- a recipe of the instruction data, training, and evaluation for long context alignment. First, we construct a long instruction-following dataset using Self-Instruct. To ensure the data diversity, it covers a broad range of tasks from various long context sources. Second, we adopt the packing and sorted batching strategies to speed up supervised fine-tuning on data with varied length distributions. Additionally, we develop a loss weighting method to balance the contribution to the loss across different sequences during packing training. Third, we introduce the LongBench-Chat benchmark for evaluating instruction-following capabilities on queries of 10k-100k in length. Experiments show that LongAlign outperforms existing recipes for LLMs in long context tasks by up to 30%, while also maintaining their proficiency in handling short, generic tasks. The code, data, and long-aligned models are open-sourced at https://github.com/THUDM/LongAlign.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 50 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

35 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 32

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
20 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2024Mixtral of Expertscited by 1,823
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  7. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  9. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  10. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  11. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  12. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  13. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  14. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  15. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  16. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  17. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  18. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  19. 2020CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Modelcited by 36
  20. 2024Orion-14B: Open-source Multilingual Large Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 29; 2025: 46; 2026: 31.

106 papers.

Sort:

By area:

  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  2. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  3. 2024Long Context Transfer from Language to Visioncited by 515
  4. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  5. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
  6. 2025Qwen2.5-1M Technical Reportcited by 164
  7. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  8. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  9. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 135
  10. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 128
  11. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 127
  12. 2024Make Your LLM Fully Utilize the Contextcited by 119
38 more
  1. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 111
  2. 2024LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QAcited by 81
  3. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 64
  4. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  5. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 62
  6. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 56
  7. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 53
  8. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 52
  9. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49
  10. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  11. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  12. 2024LongReward: Improving Long-context Large Language Models with AI Feedbackcited by 34
  13. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 32
  14. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 31
  15. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  16. 2025LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimizationcited by 24
  17. 2024Long Context Alignment with Short Instructions and Synthesized Positionscited by 22
  18. 2025Issue Localization via LLM-Driven Iterative Code Graph Searchingcited by 22
  19. 2025SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Modelscited by 20
  20. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  21. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inferencecited by 18
  22. 2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
  23. 2025Hierarchical Document Refinement for Long-context Retrieval-augmented Generationcited by 16
  24. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  25. 2026LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewardscited by 15
  26. 2024What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practicescited by 14
  27. 2025Shifting Long-Context LLMs Research from Input to Outputcited by 14
  28. 2025VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?cited by 11
  29. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  30. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 10
  31. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  32. 2025SurveyG: A Multi-Agent LLM Framework with Hierarchical Citation Graph for Automated Survey Generationcited by 10
  33. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Modelscited by 10
  34. 2024ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Modelscited by 9
  35. 2024LOGO -- Long cOntext aliGnment via efficient preference Optimizationcited by 9
  36. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  37. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  38. 2025ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learningcited by 9

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMs
  2. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
  3. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  4. 2024Efficient Alignment of Large Language Models via Data Sampling
  5. 2024GATEAU: Selecting Influential Samples for Long Context Alignment
  6. 2024CodecLM: Aligning Language Models with Tailored Synthetic Data
  7. 2025Towards Efficient and Effective Alignment of Large Language Models
  8. 2025A Survey on Training-free Alignment of Large Language Models