mlsanityData to 14 Oct 2026

QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

arXiv 2606.03968 ↗First posted 2 Jun 2026cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Rubric-based RL is a promising route for extending reinforcement learning beyond verifiable rewards, yet existing methods optimize rubrics while treating the query distribution as fixed. We identify a structural bottleneck: rubric quality is constrained by query structure. Open-ended queries yield vague rubrics; naively narrowing them introduces fabricated references that no model can verify, so all responses fail and training receives no reward signal. We present QUBRIC, a framework that co-designs queries and rubrics. Teacher-derived key points ground the rewriting of open-ended queries into scenario-based, evaluable questions. Contrastive rubric generation then turns teacher-policy gaps into query-level criteria, and learnability filtering retains only informative query-rubric pairs for GRPO training. QUBRIC achieves a +5.5 point gain on ArenaHard over the SFT baseline. Trained only on instruction-following data, it further transfers to three held-out benchmarks spanning legal, moral, and narrative reasoning (+6.3 points on average), with improvements concentrated in reasoning-related dimensions. These results provide evidence that co-designing queries and rubrics can make rubric-based RL a practical complement to RLVR beyond strictly verifiable tasks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 27 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

26 of 27 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  7. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  8. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  9. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  10. 2022Defining and Characterizing Reward Hackingcited by 203
  11. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 201
  12. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
14 more
  1. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  2. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 114
  3. 2025Reinforcement Learning with Rubric Anchorscited by 113
  4. 2024Rule Based Rewards for Language Model Safetycited by 108
  5. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  6. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  7. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  8. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  9. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  10. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 23
  11. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 15
  12. 2026Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Textcited by 10
  13. 2025Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoningcited by 6
  14. 2026From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Rufus-Air: An Open LLM Post-Training Recipecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
  2. 2026ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning
  3. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
  4. 2026Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics
  5. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Tasks
  6. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
  7. 2026Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance
  8. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR