mlsanityData to 14 Oct 2026

Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation

arXiv 2506.15068 ↗First posted 18 Jun 2025cs.CL cs.LGcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Evaluating open-ended long-form generation is challenging because it is hard to define what clearly separates good from bad outputs. Existing methods often miss key aspects like coherence, style, or relevance, or are biased by pretraining data, making open-ended long-form evaluation an underexplored problem. To address this gap, we propose PrefBERT, a scoring model for evaluating open-ended long-form generation in GRPO and guiding its training with distinct rewards for good and bad outputs. Trained on two response evaluation datasets with diverse long-form styles and Likert-rated quality, PrefBERT effectively supports GRPO by offering better semantic reward feedback than traditional metrics ROUGE-L and BERTScore do. Through comprehensive evaluations, including LLM-as-a-judge, human ratings, and qualitative analysis, we show that PrefBERT, trained on multi-sentence and paragraph-length responses, remains reliable across varied long passages and aligns well with the verifiable rewards GRPO needs. Human evaluations confirm that using PrefBERT as the reward signal to train policy models yields responses better aligned with human preferences than those trained with traditional metrics. Our code is available at https://github.com/zli12321/long_form_rl.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 53 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

37 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data21 May
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  3. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  4. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment30 Sep
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2024A Survey on LLM-as-a-Judgecited by 1,423
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  12. 2023LIMA: Less Is More for Alignmentcited by 1,100
20 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  2. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  3. 2019ELI5: Long Form Question Answeringcited by 575
  4. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  5. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  6. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  7. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  8. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  9. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  10. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Modelscited by 82
  11. 2019A Survey of Deep Reinforcement Learning in Video Gamescited by 73
  12. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  13. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
  14. 2023LongForm: Effective Instruction Tuning with Reverse Instructionscited by 47
  15. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  16. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  17. 2024PEDANTS: Cheap but Effective and Interpretable Answer Equivalencecited by 31
  18. 2024Enhancing Text Classification through LLM-Driven Active Learning and Human Annotationcited by 20
  19. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEscited by 13
  20. 2024Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarioscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  2. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 97
  3. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 35
  4. 2025Language Models that Think, Chat Bettercited by 24
  5. 2025SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimizationcited by 12
  6. 2025RAVR: Reference-Answer-guided Variational Reasoning for Large Language Modelscited by 8
  7. 2026EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimizationcited by 5
  8. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  9. 2025Compositional Coordination for Multi-Robot Teams with Large Language Modelscited by 3
  10. 2025Unleashing Scientific Reasoning for Bio-experimental Protocol Generation via Structured Component-based Reward Mechanismcited by 3
  11. 2026R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenGenAlign: A Preference Dataset and Benchmark for Trustworthy Reward Modeling in Open-Ended, Long-Context Generation
  2. 2026ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training
  3. 2026EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
  4. 2025Shaping Explanations: Semantic Reward Modeling with Encoder-Only Transformers for GRPO
  5. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
  6. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models
  7. 2024Self-Generated Critiques Boost Reward Modeling for Language Models
  8. 2025OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning