# V0: A Generalist Value Model for Any Policy at State Zero

arXiv: https://arxiv.org/abs/2602.03584 · HTML: https://mlsanity.com/paper/2602.03584/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 45 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 30 of 37)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks](https://mlsanity.com/paper/2504.05118/) (2025)
- [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [AlphaMath Almost Zero: Process Supervision without Process](https://mlsanity.com/paper/2405.03553/) (2024)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts](https://mlsanity.com/paper/2506.02177/) (2025)
- [What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret](https://mlsanity.com/paper/2503.01491/) (2025)
- [Vision Language Models are In-Context Value Learners](https://mlsanity.com/paper/2411.04549/) (2024)
- [RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models](https://mlsanity.com/paper/2409.19886/) (2024)
- [Universal Model Routing for Efficient LLM Inference](https://mlsanity.com/paper/2502.08773/) (2025)
- [LongCat-Flash Technical Report](https://mlsanity.com/paper/2509.01322/) (2025)
- [Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay](https://mlsanity.com/paper/2506.05316/) (2025)
- [Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?](https://mlsanity.com/paper/2510.01161/) (2025)
- [Value Augmented Sampling for Language Model Alignment and Personalization](https://mlsanity.com/paper/2405.06639/) (2024)
- [EmbedLLM: Learning Compact Representations of Large Language Models](https://mlsanity.com/paper/2410.02223/) (2024)
- [Why Larger Language Models Do In-context Learning Differently?](https://mlsanity.com/paper/2405.19592/) (2024)
- [Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration](https://mlsanity.com/paper/2508.13755/) (2025)
- [The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants](https://mlsanity.com/paper/2505.19797/) (2025)
- [Generalist Reward Models: Found Inside Large Language Models](https://mlsanity.com/paper/2506.23235/) (2025)
- [Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing](https://mlsanity.com/paper/2508.12631/) (2025)
- [Efficient Reinforcement Learning with Large Language Model Priors](https://mlsanity.com/paper/2410.07927/) (2024)

## Built on by (most cited first, 7 of 7)

- [Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning](https://mlsanity.com/paper/2604.10701/) (2026)
- [V0.5: Generalist Value Model as a Prior for Sparse RL Rollouts](https://mlsanity.com/paper/2603.10848/) (2026)
- [CAST: Game Solvers as Turn-Level Teachers for LLM Agents](https://mlsanity.com/paper/2607.25308/) (2026)
- [SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL](https://mlsanity.com/paper/2608.24870/) (2026)
- [Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling](https://mlsanity.com/paper/2604.27039/) (2026)
- [Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR](https://mlsanity.com/paper/2609.37825/) (2026)
- [T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning](https://mlsanity.com/paper/2610.00388/) (2026)

## Abstract (arXiv)

Policy gradient methods rely on a baseline to measure the relative advantage of an action, ensuring the model reinforces behaviors that outperform its current average capability. In the training of Large Language Models (LLMs) using Actor-Critic methods (e.g., PPO), this baseline is typically estimated by a Value Model (Critic) often as large as the policy model itself. However, as the policy continuously evolves, the value model requires expensive, synchronous incremental training to accurately track the shifting capabilities of the policy. To avoid this overhead, Group Relative Policy Optimization (GRPO) eliminates the coupled value model by using the average reward of a group of rollouts as the baseline; yet, this approach necessitates extensive sampling to maintain estimation stability. In this paper, we propose V0, a Generalist Value Model capable of estimating the expected performance of any model on unseen prompts without requiring parameter updates. We reframe value estimation by treating the policy's dynamic capability as an explicit context input; specifically, we leverage a history of instruction-performance pairs to dynamically profile the model, departing from the traditional paradigm that relies on parameter fitting to perceive capability shifts. Focusing on value estimation at State Zero (i.e., the initial prompt, hence V0), our model serves as a critical resource scheduler. During GRPO training, V0 predicts success rates prior to rollout, allowing for efficient sampling budget allocation; during deployment, it functions as a router, dispatching instructions to the most cost-effective and suitable model. Empirical results demonstrate that V0 significantly outperforms heuristic budget allocation and achieves a Pareto-optimal trade-off between performance and cost in LLM routing tasks.