mlsanityData to 13 Oct 2026

Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning

arXiv 2502.13820 ↗First posted 19 Feb 2025cs.AI cs.CL cs.LGcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Synthetic verification techniques such as generating test cases and reward modelling are common ways to enhance the coding capabilities of large language models (LLM) beyond predefined tests. Additionally, code verification has recently found great success as a critical component in improving reasoning capability of LLMs via reinforcement learning. In this paper, we propose an approach which can transform existing coding benchmarks into scoring and ranking datasets to evaluate the effectiveness of synthetic verifiers. We also propose multiple metrics to measure different aspects of the synthetic verifiers with the proposed benchmarks. By employing the proposed approach, we release four new benchmarks (HE-R, HE-R+, MBPP-R, and MBPP-R+), and analyzed synthetic verification methods with standard, reasoning-based, and reward-based LLMs. Our experiments show that reasoning can significantly improve test case generation and that scaling the number of test cases enhances the verification accuracy.

Key facts. Cited by 9 papers in the map (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 37 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

33 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug

Other references · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,486
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  3. 2024Qwen2.5 Technical Reportcited by 4,840
  4. 2024GPT-4o System Cardcited by 4,536
  5. 2021Program Synthesis with Large Language Modelscited by 3,573
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,362
  7. 2024OpenAI o1 System Cardcited by 1,233
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,179
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 488
  10. 2022CodeT: Code Generation with Generated Testscited by 377
  11. 2024Nemotron-4 340B Technical Reportcited by 144
  12. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 130
19 more
  1. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 112
  2. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 101
  3. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 91
  4. 2024Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLMcited by 64
  5. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  6. 2024TestGenEval: A Real World Unit Test Generation and Test Completion Benchmarkcited by 49
  7. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
  8. 2024TESTEVAL: Benchmarking Large Language Models for Test Case Generationcited by 38
  9. 2024TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?cited by 32
  10. 2025Dynamic Scaling of Unit Tests for Code Reward Modelingcited by 31
  11. 2024CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?cited by 29
  12. 2024Large Language Models as Test Case Generators: Performance Evaluation and Enhancementcited by 27
  13. 2025Process-Supervised Reinforcement Learning for Code Generationcited by 22
  14. 2025COFFE: A Code Efficiency Benchmark for Code Generationcited by 19
  15. 2024TestBench: Evaluating Class-Level Test Case Generation Capability of Large Language Modelscited by 15
  16. 2024Measuring the Influence of Incorrect Code on Test Generationcited by 11
  17. 2024Scattered Forest Search: Smarter Code Space Exploration with LLMscited by 10
  18. 2024DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMscited by 9
  19. 2024Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropycited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 58
  2. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  3. 2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critiquecited by 24
  4. 2025MARS: toward more efficient multi-agent collaboration for LLM reasoningcited by 13
  5. 2026Auditing Reward Hackability in Code RL Training Environmentscited by 3
  6. 2026Learning Generative Selection for Best-of-Ncited by 2
  7. 2026Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Setscited by 2
  8. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 0
  9. 2026Learning What to Skip: Counterfactual Credit Assignment for Efficient Multi-Agent LLM Workflowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HardTests: Synthesizing High-Quality Test Cases for LLM Coding
  2. 2025Rethinking Verification for LLM Code Generation: From Generation to Testing
  3. 2025VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains
  4. 2025When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers
  5. 2026Learning from Synthetic Data Improves Multi-hop Reasoning
  6. 2026LLM-as-a-Verifier: A General-Purpose Verification Framework
  7. 2026Automating Formal Verification with Reinforcement Learning and Recursive Inference
  8. 2025CodeContests+: High-Quality Test Case Generation for Competitive Programming