# CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward

arXiv: https://arxiv.org/abs/2508.03686 · HTML: https://mlsanity.com/paper/2508.03686/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 51 of its 68 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 30 of 51)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Reinforcement Learning for Reasoning in Large Language Models with One Training Example](https://mlsanity.com/paper/2504.20571/) (2025)
- [ProcessBench: Identifying Process Errors in Mathematical Reasoning](https://mlsanity.com/paper/2412.06559/) (2024)
- [ReFT: Reasoning with Reinforced Fine-Tuning](https://mlsanity.com/paper/2401.08967/) (2024)
- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [V-STaR: Training Verifiers for Self-Taught Reasoners](https://mlsanity.com/paper/2402.06457/) (2024)

## Built on by (most cited first, 30 of 40)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction](https://mlsanity.com/paper/2604.27393/) (2026)
- [Intern-S1: A Scientific Multimodal Foundation Model](https://mlsanity.com/paper/2508.15763/) (2025)
- [Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers](https://mlsanity.com/paper/2510.00915/) (2025)
- [MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods](https://mlsanity.com/paper/2601.21821/) (2026)
- [Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities](https://mlsanity.com/paper/2602.05073/) (2026)
- [Variation in Verification: Understanding Verification Dynamics in Large Language Models](https://mlsanity.com/paper/2509.17995/) (2025)
- [Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning](https://mlsanity.com/paper/2510.23038/) (2025)
- [Agent-as-a-Judge](https://mlsanity.com/paper/2601.05111/) (2026)
- [Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation](https://mlsanity.com/paper/2510.04265/) (2025)
- [Future-as-Label: Scalable Supervision from Real-World Outcomes](https://mlsanity.com/paper/2601.06336/) (2026)
- [ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch](https://mlsanity.com/paper/2601.13606/) (2026)
- [STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens](https://mlsanity.com/paper/2602.15620/) (2026)
- [FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning](https://mlsanity.com/paper/2510.22543/) (2025)
- [The Era of Agentic Organization: Learning to Organize with Language Models](https://mlsanity.com/paper/2510.26658/) (2025)
- [SCI-Verifier: Scientific Verifier with Thinking](https://mlsanity.com/paper/2509.24285/) (2025)
- [ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning](https://mlsanity.com/paper/2511.14366/) (2025)
- [Aletheia: What Makes RLVR For Code Verifiers Tick?](https://mlsanity.com/paper/2601.12186/) (2026)
- [CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questions](https://mlsanity.com/paper/2512.01224/) (2025)
- [OPV: Outcome-based Process Verifier for Efficient Long Chain-of-Thought Verification](https://mlsanity.com/paper/2512.10756/) (2025)
- [Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasets](https://mlsanity.com/paper/2601.09733/) (2025)
- [Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Training](https://mlsanity.com/paper/2603.07223/) (2026)
- [LogiNumSynth: Synthesizing Joint Logical-Numerical Reasoning Problems for Language Models](https://mlsanity.com/paper/2510.11031/) (2025)
- [Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rollouts](https://mlsanity.com/paper/2609.06386/) (2026)
- [Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling](https://mlsanity.com/paper/2602.06795/) (2026)
- [PRIME: A Process-Outcome Alignment Benchmark for Verifiable Reasoning in Mathematics and Engineering](https://mlsanity.com/paper/2602.11570/) (2026)
- [ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework](https://mlsanity.com/paper/2604.07506/) (2026)
- [ScaleBox: Enabling High-Fidelity and Scalable Code Verification for Large Language Models](https://mlsanity.com/paper/2604.27467/) (2026)
- [MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning](https://mlsanity.com/paper/2610.01352/) (2026)
- [CharTool: Tool-Integrated Visual Reasoning for Chart Understanding](https://mlsanity.com/paper/2604.02794/) (2026)

## Abstract (arXiv)

Answer verification is crucial not only for evaluating large language models (LLMs) by matching their unstructured outputs against standard answers, but also serves as the reward model to guide LLM optimization. Most evaluation frameworks rely on regularized matching or employ general LLMs for answer verification, which demands extensive, repetitive customization for regex rules or evaluation prompts. Two fundamental limitations persist in current methodologies: 1) the absence of comprehensive benchmarks that systematically evaluate verification capabilities across different LLMs; and 2) the nascent stage of verifier development, where existing approaches lack both the robustness to handle complex edge cases and the generalizability across different domains. In this work, we develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward. It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types, including multi-subproblems, formulas, and sequence answers, while effectively identifying abnormal/invalid responses. We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier. We anticipate that CompassVerifier and VerifierBench will facilitate answer verification, evaluation protocols, and reinforcement learning research. Code and dataset are available at https://github.com/open-compass/CompassVerifier.