# Soft-SVeRL: Self-Verified Reinforcement Learning with Soft Rewards

arXiv: https://arxiv.org/abs/2605.28561 · HTML: https://mlsanity.com/paper/2605.28561/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 14 of 14)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [RLPR: Extrapolating RLVR to General Domains without Verifiers](https://mlsanity.com/paper/2506.18254/) (2025)
- [Reinforcing General Reasoning without Verifiers](https://mlsanity.com/paper/2505.21493/) (2025)
- [Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning](https://mlsanity.com/paper/2506.03136/) (2025)
- [Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding](https://mlsanity.com/paper/2411.04282/) (2024)
- [Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards](https://mlsanity.com/paper/2506.00103/) (2025)
- [Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data](https://mlsanity.com/paper/2503.19618/) (2025)
- [Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning](https://mlsanity.com/paper/2504.08672/) (2025)
- [Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision](https://mlsanity.com/paper/2509.14234/) (2025)
- [Scalable Training of Language Models using JAX pjit and TPUv4](https://mlsanity.com/paper/2204.06514/) (2022)

## Built on by (most cited first, 1 of 1)

- [Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck](https://mlsanity.com/paper/2608.20362/) (2026)

## Abstract (arXiv)

Reinforcement Learning from Verifiable Rewards (RLVR) has improved language models in domains such as mathematics and code, where correctness can be checked automatically. However, many important tasks are only partially verifiable: prompts contain multiple requirements, responses may satisfy some but not all of them, or no single reference answer might exist. We introduce Soft-RLVR, a framework for reinforcement learning from decomposed, learned verification signals. Soft-RLVR converts each prompt into a checklist of atomic requirements, scores candidate responses item by item with an LLM verifier, and trains on the resulting soft reward. Checklist-based rewards turn sparse pass/fail supervision into a denser partial-credit signal, but they also introduce a tradeoff: averaging item-level judgments can reduce verifier noise, while partial credit can reward incomplete responses. We formalize this tradeoff and identify conditions under which checklist-based verification gives a more reliable RL training signal than holistic verification. We further introduce Soft-SVeRL, a self-verifying variant of Soft-RLVR in which the policy also acts as the verifier. We show that self-verification is prone to reward inflation from overly permissive self-judgments, and that explicit stabilization is needed to prevent this collapse. In a controlled instruction-following setting with rule-based ground-truth evaluation, checklist-based Soft-RLVR improves IFEval by up to 11.1 points using only learned verifier rewards. Our experiments further show that verifier quality and checklist quality both affect downstream RL outcomes, and that explicit stabilization is essential for effective self-verification.