mlsanityData to 14 Oct 2026

A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards

arXiv 2609.33467 ↗First posted 27 Sep 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

When post-training large language models on tasks with semi-verifiable rewards, there are many factors (training steps, base model size, training order, data quality, verifier accuracy, etc.) that practitioners must contend with to maximize model performance. Yet, it remains unclear how well verifier agreement predicts post-training performance on such tasks. In this paper, we explore this question with over 11k H100 GPU-hours, across HealthBench and PRBench tasks in medical, legal, and finance domains. Across the tested domains, Qwen3 trainees (1.7B-8B on HealthBench; 8B on PRBench), evaluation splits, and frontier LLM reference judges (which we call golden verifiers), higher verifier agreement does not consistently identify the best training verifier. Expensive verifiers need not outperform inexpensive ones, and open-weight Gemma verifiers produce strong training outcomes. We compare two low-cost choices retrospectively -- a cost-reducing choice and a balanced choice -- with estimated grading cost reductions of 98.8%-99.7% relative to the golden grading protocols and average post-training score gaps of 1-3 points from the best evaluated training verifier. These averages include larger losses in individual settings; they do not establish that verifier choices are interchangeable.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

29 of 38 references.

References · 29

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  7. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  8. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  9. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  10. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  11. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  12. 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 158
17 more
  1. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 156
  2. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  3. 2026DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compressioncited by 76
  4. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  5. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 51
  6. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  7. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 45
  8. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 32
  9. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 31
  10. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 27
  11. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 20
  12. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 13
  13. 2026Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Trainingcited by 13
  14. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 13
  15. 2025Tuning LLM Judge Design Decisions for 1/1000 of the Costcited by 11
  16. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimizationcited by 11
  17. 2026BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflowscited by 2

Built on by

Not cited yet.