# LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization

arXiv: https://arxiv.org/abs/2502.13922 · HTML: https://mlsanity.com/paper/2502.13922/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [Iterative Reasoning Preference Optimization](https://mlsanity.com/paper/2404.19733/) (2024)
- [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://mlsanity.com/paper/2309.14509/) (2023)
- [Self-Alignment with Instruction Backtranslation](https://mlsanity.com/paper/2308.06259/) (2023)
- [LongAlign: A Recipe for Long Context Alignment of Large Language Models](https://mlsanity.com/paper/2401.18058/) (2024)
- [CLEX: Continuous Length Extrapolation for Large Language Models](https://mlsanity.com/paper/2310.16450/) (2023)
- [Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation](https://mlsanity.com/paper/2402.11907/) (2024)

## Built on by (most cited first, 24 of 24)

- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [OPSDL: On-Policy Self-Distillation for Long-Context Language Models](https://mlsanity.com/paper/2604.17535/) (2026)
- [LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation](https://mlsanity.com/paper/2509.09754/) (2025)
- [LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards](https://mlsanity.com/paper/2603.02146/) (2026)
- [LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs](https://mlsanity.com/paper/2503.02502/) (2025)
- [SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization](https://mlsanity.com/paper/2505.11166/) (2025)
- [WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale](https://mlsanity.com/paper/2502.16684/) (2025)
- [Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization](https://mlsanity.com/paper/2502.16825/) (2025)
- [When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework](https://mlsanity.com/paper/2506.16411/) (2025)
- [Self-Improving Model Steering](https://mlsanity.com/paper/2507.08967/) (2025)
- [Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimization](https://mlsanity.com/paper/2508.13993/) (2025)
- [Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shaping](https://mlsanity.com/paper/2601.12465/) (2026)
- [How to Train Your Long-Context Visual Document Model](https://mlsanity.com/paper/2602.15257/) (2026)
- [DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models](https://mlsanity.com/paper/2602.22175/) (2026)
- [Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation](https://mlsanity.com/paper/2509.01185/) (2025)
- [Policy Optimization Prefers The Path of Least Resistance](https://mlsanity.com/paper/2510.21853/) (2025)
- [Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contexts](https://mlsanity.com/paper/2512.03704/) (2025)
- [Autoregressive Direct Preference Optimization](https://mlsanity.com/paper/2602.09533/) (2026)
- [Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty](https://mlsanity.com/paper/2510.05534/) (2025)
- [Do Implicit Personalization and Explicit Styles Conflict? PsPLUG: A Lightweight Plug-in for Balancing Personalization and Style in Customized LLMs](https://mlsanity.com/paper/2601.06362/) (2026)
- [Document Reconstruction Unlocks Scalable Long-Context RLVR](https://mlsanity.com/paper/2602.08237/) (2026)
- [Internalized Reasoning for Long-Context Visual Document Understanding](https://mlsanity.com/paper/2604.02371/) (2026)
- [Unsupervised Post-Training of Foundation Models: A Survey](https://mlsanity.com/paper/2608.24982/) (2026)
- [Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakes](https://mlsanity.com/paper/2609.19156/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities through pretraining and alignment. However, superior short-context LLMs may underperform in long-context scenarios due to insufficient long-context alignment. This alignment process remains challenging due to the impracticality of human annotation for extended contexts and the difficulty in balancing short- and long-context performance. To address these challenges, we introduce LongPO, that enables short-context LLMs to self-evolve to excel on long-context tasks by internally transferring short-context capabilities. LongPO harnesses LLMs to learn from self-generated short-to-long preference data, comprising paired responses generated for identical instructions with long-context inputs and their compressed short-context counterparts, respectively. This preference reveals capabilities and potentials of LLMs cultivated during short-context alignment that may be diminished in under-aligned long-context scenarios. Additionally, LongPO incorporates a short-to-long KL constraint to mitigate short-context performance decline during long-context alignment. When applied to Mistral-7B-Instruct-v0.2 from 128K to 512K context lengths, LongPO fully retains short-context performance and largely outperforms naive SFT and DPO in both long- and short-context tasks. Specifically, LongPO-trained models can achieve results on long-context benchmarks comparable to, or even surpassing, those of superior LLMs (e.g., GPT-4-128K) that involve extensive long-context annotation and larger parameter scales. Our code is available at https://github.com/DAMO-NLP-SG/LongPO.