# Large Language Models Can Self-Improve in Long-context Reasoning

arXiv: https://arxiv.org/abs/2411.08147 · HTML: https://mlsanity.com/paper/2411.08147/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 48 of its 110 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)
- Synthetic data and self-improvement for LLMs / [Synthetic reasoning data and bootstrapping](https://mlsanity.com/synth/synthetic-reasoning-data-and-bootstrapping/)

## Builds on (most cited first, 30 of 48)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://mlsanity.com/paper/2309.14509/) (2023)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [LongNet: Scaling Transformers to 1,000,000,000 Tokens](https://mlsanity.com/paper/2307.02486/) (2023)
- [Data Engineering for Scaling Language Models to 128K Context](https://mlsanity.com/paper/2402.10171/) (2024)
- [The Capacity for Moral Self-Correction in Large Language Models](https://mlsanity.com/paper/2302.07459/) (2023)
- [How to Train Long-Context Language Models (Effectively)](https://mlsanity.com/paper/2410.02660/) (2024)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [jina-embeddings-v3: Multilingual Embeddings With Task LoRA](https://mlsanity.com/paper/2409.10173/) (2024)
- [Evaluation of OpenAI o1: Opportunities and Challenges of AGI](https://mlsanity.com/paper/2409.18486/) (2024)
- [HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly](https://mlsanity.com/paper/2410.02694/) (2024)
- [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://mlsanity.com/paper/2406.17419/) (2024)

## Built on by (most cited first, 29 of 29)

- [Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization](https://mlsanity.com/paper/2504.05812/) (2025)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning](https://mlsanity.com/paper/2502.04780/) (2025)
- [Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.24377/) (2025)
- [100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models](https://mlsanity.com/paper/2505.00551/) (2025)
- [LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts](https://mlsanity.com/paper/2510.19363/) (2025)
- [Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervision](https://mlsanity.com/paper/2502.20790/) (2025)
- [LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data](https://mlsanity.com/paper/2502.12583/) (2025)
- [SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization](https://mlsanity.com/paper/2511.17938/) (2025)
- [Self-Taught Agentic Long Context Understanding](https://mlsanity.com/paper/2502.15920/) (2025)
- [Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs](https://mlsanity.com/paper/2503.23219/) (2025)
- [Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillation](https://mlsanity.com/paper/2508.15709/) (2025)
- [Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models](https://mlsanity.com/paper/2502.18817/) (2025)
- [Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognition](https://mlsanity.com/paper/2503.12303/) (2025)
- [Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check](https://mlsanity.com/paper/2509.11629/) (2025)
- [Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrieval](https://mlsanity.com/paper/2503.09819/) (2025)
- [Revisiting Long-context Modeling from Context Denoising Perspective](https://mlsanity.com/paper/2510.05862/) (2025)
- [CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability](https://mlsanity.com/paper/2505.10063/) (2025)
- [Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimization](https://mlsanity.com/paper/2508.13993/) (2025)
- [Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shaping](https://mlsanity.com/paper/2601.12465/) (2026)
- [Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoning](https://mlsanity.com/paper/2604.21327/) (2026)
- [Flora: Effortless Context Construction to Arbitrary Length and Scale](https://mlsanity.com/paper/2507.19786/) (2025)
- [LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards](https://mlsanity.com/paper/2605.31584/) (2026)
- [OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling](https://mlsanity.com/paper/2608.05141/) (2026)
- [Probing How Scalable Table Data Enhances General Long-Context Reasoning](https://mlsanity.com/paper/2603.21719/) (2026)
- [A Decomposition Perspective to Long-context Reasoning for LLMs](https://mlsanity.com/paper/2604.07981/) (2026)
- [Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning](https://mlsanity.com/paper/2606.18831/) (2026)
- [Unsupervised Post-Training of Foundation Models: A Survey](https://mlsanity.com/paper/2608.24982/) (2026)
- [StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2609.38809/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved substantial progress in processing long contexts but still struggle with long-context reasoning. Existing approaches typically involve fine-tuning LLMs with synthetic data, which depends on annotations from human experts or advanced models like GPT-4, thus restricting further advancements. To address this issue, we investigate the potential for LLMs to self-improve in long-context reasoning and propose ours, an approach specifically designed for this purpose. This approach is straightforward: we sample multiple outputs for each question, score them with Minimum Bayes Risk, and then apply supervised fine-tuning or preference optimization based on these outputs. Extensive experiments on several leading LLMs demonstrate the effectiveness of ours, with an absolute improvement of 4.2 points for Llama-3.1-8B-Instruct. Furthermore, ours achieves superior performance compared to prior approaches that depend on data produced by human experts or advanced models. We anticipate that this work will open new avenues for self-improvement techniques in long-context scenarios, which are essential for the continual advancement of LLMs.