mlsanityData to 7 Oct 2026

RankBuffer: Efficient Ranking-Based Rewards for Open-Ended Generation

arXiv 2609.36652 ↗First posted 29 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Open-ended generation lacks canonical answers, making pointwise rewards difficult to calibrate for group-based reinforcement learning. Directly ranking same-query rollouts provides a more suitable relative reward signal, but existing ranking-based reward methods can incur substantial judging cost. We introduce RankBuffer, which maintains an ordered, query-specific buffer of previously judged responses as a reusable quality scale. Each rollout is first inserted into an anchor interval through an independent coarse judgment, after which only rollouts assigned to the same interval undergo local fine ranking. The resulting complete order is converted into bounded rank rewards, while boundary expansion, local refinement, and inactive-anchor pruning adapt the buffer as the policy evolves. Across four open-ended benchmarks, RankBuffer consistently outperforms all pointwise baselines. It also achieves nearly on-par performance with the strongest ranking-based reward baseline while substantially reducing judging cost. Ablations demonstrate the importance of both local fine ranking and anchor response content, while buffer analyses show that rollout-derived anchors progressively extend and refine the covered quality scale. These results establish response reuse as an effective approach to efficient relative reward construction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 42 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

33 of 42 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  10. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
21 more
  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  2. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  3. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  4. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  5. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  6. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  7. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  8. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  9. 2025Reinforcement Learning with Rubric Anchorscited by 108
  10. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 95
  11. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  12. 2024TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategycited by 47
  13. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  14. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  15. 2024Leveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Modelscited by 20
  16. 2026ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Rankingcited by 17
  17. 2026Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judgecited by 16
  18. 2026EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learningcited by 11
  19. 2026OASES: Outcome-Aligned Search-Evaluation Co-Training for Agentic Searchcited by 7
  20. 2026Tournament-GRPO: Group-Wise Tournament Rewards for Reinforcement Learning in Open-Ended Long-Form Generationcited by 7
  21. 2026RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Constructioncited by 2

Built on by

Not cited yet.