mlsanityData to 14 Oct 2026

From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation

arXiv 2506.16024 ↗First posted 19 Jun 2025cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Current research on long-form context in Large Language Models (LLMs) primarily focuses on the understanding of long-contexts, the Open-ended Long Text Generation (Open-LTG) remains insufficiently explored. Training a long-context generation model requires curation of gold standard reference data, which is typically nonexistent for informative Open-LTG tasks. However, previous methods only utilize general assessments as reward signals, which limits accuracy. To bridge this gap, we introduce ProxyReward, an innovative reinforcement learning (RL) based framework, which includes a dataset and a reward signal computation method. Firstly, ProxyReward Dataset generation is accomplished through simple prompts that enables the model to create automatically, obviating extensive labeled data or significant manual effort. Secondly, ProxyReward Signal offers a targeted evaluation of information comprehensiveness and accuracy for specific questions. The experimental results indicate that our method ProxyReward surpasses even GPT-4-Turbo. It can significantly enhance performance by 20% on the Open-LTG task when training widely used open-source models, while also surpassing the LLM-as-a-Judge approach. Our work presents effective methods to enhance the ability of LLMs to address complex open-ended questions posed by human.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 33 of its 54 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

33 of 54 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  7. 2024A Survey on LLM-as-a-Judgecited by 1,423
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  10. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  11. 2024Self-Rewarding Language Modelscited by 681
  12. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
21 more
  1. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  2. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  4. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  5. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  6. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  7. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  8. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  9. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  10. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  11. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  12. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  13. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  14. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  15. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  16. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  17. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 55
  18. 2023LongForm: Effective Instruction Tuning with Reverse Instructionscited by 47
  19. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  20. 2024Suri: Multi-constraint Instruction Following for Long-form Text Generationcited by 34
  21. 2024Exploration of Masked and Causal Language Modelling for Text Generationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Understanding and Mitigating Political Stance Cross-topic Generalization in Large Language Modelscited by 6
  3. 2025Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Followingcited by 6
  4. 2026SEIF: Self-Evolving Reinforcement Learning for Instruction Followingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenGenAlign: A Preference Dataset and Benchmark for Trustworthy Reward Modeling in Open-Ended, Long-Context Generation
  2. 2024LongReward: Improving Long-context Large Language Models with AI Feedback
  3. 2025LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning
  4. 2025Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation
  5. 2026LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning
  6. 2023Prompt-Based Length Controlled Generation with Reinforcement Learning
  7. 2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards
  8. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models