mlsanityData to 7 Oct 2026

LongReward: Improving Long-context Large Language Models with AI Feedback

arXiv 2410.21252 ↗First posted 28 Oct 2024cs.CL cs.LGcited by 34

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Though significant advancements have been achieved in developing long-context large language models (LLMs), the compromised quality of LLM-synthesized data for supervised fine-tuning (SFT) often affects the long-context performance of SFT models and leads to inherent limitations. In principle, reinforcement learning (RL) with appropriate reward signals can further enhance models' capacities. However, how to obtain reliable rewards in long-context scenarios remains unexplored. To this end, we propose LongReward, a novel method that utilizes an off-the-shelf LLM to provide rewards for long-context model responses from four human-valued dimensions: helpfulness, logicality, faithfulness, and completeness, each with a carefully designed assessment pipeline. By combining LongReward and offline RL algorithm DPO, we are able to effectively improve long-context SFT models. Our experiments indicate that LongReward not only significantly improves models' long-context performance but also enhances their ability to follow short instructions. We also find that long-context DPO with LongReward and conventional short-context DPO can be used together without hurting either one's performance.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 47 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

30 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
16 more
  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  5. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  6. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  7. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  8. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  9. 2023Fine-tuning Language Models for Factualitycited by 219
  10. 2022Is GPT-3 a Good Data Annotator?cited by 208
  11. 2024Make Your LLM Fully Utilize the Contextcited by 118
  12. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  13. 2024LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QAcited by 78
  14. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Modelscited by 48
  15. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  16. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedbackcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 20; 2026: 12.

34 papers.

Sort:

By area:

  1. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  2. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  3. 2025Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Modelscited by 43
  4. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  5. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  6. 2026OPSDL: On-Policy Self-Distillation for Long-Context Language Modelscited by 29
  7. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  8. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  9. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Datacited by 16
  10. 2025A Cognitive Writing Perspective for Constrained Long-Form Text Generationcited by 14
  11. 2025LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigmcited by 14
  12. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
22 more
  1. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  2. 2025LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Informationcited by 8
  3. 2025When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Frameworkcited by 8
  4. 2025Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learningcited by 7
  5. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  6. 2026Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modelingcited by 7
  7. 2025OpenGenAlign: A Preference Dataset and Benchmark for Trustworthy Reward Modeling in Open-Ended, Long-Context Generationcited by 5
  8. 2025Revisiting Long-context Modeling from Context Denoising Perspectivecited by 5
  9. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modelingcited by 5
  10. 2026From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Modelscited by 4
  11. 2025Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimizationcited by 3
  12. 2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
  13. 2025Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generationcited by 2
  14. 2025Tree of Agents: Improving Long-Context Capabilities of Large Language Models through Multi-Perspective Reasoningcited by 2
  15. 2026DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inferencecited by 2
  16. 2026Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoningcited by 2
  17. 2026A Decomposition Perspective to Long-context Reasoning for LLMscited by 1
  18. 2026Document Reconstruction Unlocks Scalable Long-Context RLVRcited by 0
  19. 2026Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decodingcited by 0
  20. 2026Evidence-State Rewards for Long-Context Reasoningcited by 0
  21. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generationcited by 0
  22. 2026PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning
  2. 2025LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization
  3. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models
  4. 2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards
  5. 2026Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
  6. 2026LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards
  7. 2025Generalist Reward Models: Found Inside Large Language Models
  8. 2026A Recipe for Long-Context Reasoning in Large Language Models via On-Policy Optimization and Distillation