mlsanityData to 7 Oct 2026

Online Rubrics Elicitation from Pairwise Comparisons

arXiv 2510.07284 ↗First posted 8 Oct 2025cs.CL cs.AI cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

Rubrics provide a flexible way to train LLMs on open-ended long-form answers where verifiable rewards are not applicable and human preferences provide coarse signals. Prior work shows that reinforcement learning with rubric-based rewards leads to consistent gains in LLM post-training. Most existing approaches rely on rubrics that remain static over the course of training. Such static rubrics, however, are vulnerable to reward-hacking type behaviors and fail to capture emergent desiderata that arise during training. We introduce Online Rubrics Elicitation (OnlineRubrics), a method that dynamically curates evaluation criteria in an online manner through pairwise comparisons of responses from current and reference policies. This online process enables continuous identification and mitigation of errors as training proceeds. Empirically, this approach yields consistent improvements of up to 8% over training exclusively with static rubrics across AlpacaEval, GPQA, ArenaHard as well as the validation sets of expert questions and rubrics. We qualitatively analyze the elicited criteria and identify prominent themes such as transparency, practicality, organization, and reasoning.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

29 of 38 references.

References · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2024A Survey on LLM-as-a-Judgecited by 1,384
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
17 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  3. 2024InternLM2 Technical Reportcited by 391
  4. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  6. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  7. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  8. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  9. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  10. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 171
  11. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 137
  12. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
  13. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  14. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  15. 2025Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learningcited by 40
  16. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  17. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 40.

41 papers.

Sort:

By area:

  1. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  2. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  3. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 29
  4. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  5. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 18
  6. 2026Agent-as-a-Judgecited by 15
  7. 2026Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewardscited by 14
  8. 2026EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learningcited by 11
  9. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVRcited by 9
  10. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 9
  11. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 9
  12. 2026SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probingcited by 6
29 more
  1. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  2. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  3. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 4
  4. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 4
  5. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  6. 2026DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agentscited by 4
  7. 2026ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agentscited by 4
  8. 2026Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judgecited by 3
  9. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 3
  10. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 3
  11. 2026Quark Medical Alignment: A Holistic Multi-Dimensional Alignment and Collaborative Optimization Paradigmcited by 2
  12. 2026Reinforcement Learning with Robust Rubric Rewardscited by 2
  13. 2026Deep Research as Rubric for Reinforcement Learningcited by 2
  14. 2026DecoEvo: Score-Decoupled Co-Evolution of Solver and Rubric-Generator Skills in Text Spacecited by 2
  15. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RLcited by 2
  16. 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generationcited by 2
  17. 2026A Primer in Post-Training Reasoning Data: What We Know About How It Workscited by 1
  18. 2026PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perceptioncited by 1
  19. 2026Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoningcited by 1
  20. 2026DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Trainingcited by 1
  21. 2026CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judgingcited by 0
  22. 2026GrowLoop: Self-Evolving Conversation Evaluation Seeded by Humancited by 0
  23. 2026Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Shortcited by 0
  24. 2026Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidencecited by 0
  25. 2026CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Datacited by 0
  26. 2026Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMscited by 0
  27. 2026CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Trainingcited by 0
  28. 2026ARISE: Adapting to Evolving Capability Gaps in Agentic Reinforcement Learningcited by 0
  29. 2026RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillationcited by 0