mlsanityData to 7 Oct 2026

Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data

arXiv 2503.19618 ↗First posted 25 Mar 2025cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

We propose to scale RL to unverifiable data with a novel algorithm JEPO (Jensen's Evidence lower bound Policy Optimization). While most prior efforts on scaling RL for LLMs focus on verifiable data where ground truth answers are typically short-form and can be matched easily; we investigate the case where such assumptions are less valid (e.g., when answers are long-form such as mathematical proofs). To scale RL training to unverifiable data with contemporary training constraints, we propose JEPO. JEPO applies Jensen's evidence lower bound, a pragmatic simplification of the evidence lower bound which views chain-of-thought as a latent variable in the generative process. We show that on verifiable data (math), JEPO is as effective as RL with verifiable rewards; on semi-verifiable data (numina), JEPO improves on soft-match based evaluations compared to RL with verifiable rewards which can only leverage a subset of the data source; finally, on unverifiable data (numina-proof), JEPO outperforms SFT and a few ablation baselines on likelihood evaluations.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

44 of 67 references.

References · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2023Let's Verify Step by Stepcited by 3,598
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
32 more
  1. 2016Variational Inference: A Review for Statisticianscited by 1,466
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  3. 2017Hindsight Experience Replaycited by 1,096
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  7. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  8. 2024Self-Rewarding Language Modelscited by 654
  9. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  11. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  13. 2015Importance Weighted Autoencoderscited by 454
  14. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  15. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  16. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  17. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  18. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  19. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  20. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  21. 2015Hierarchical Variational Modelscited by 89
  22. 2016Variational inference for Monte Carlo objectivescited by 83
  23. 2018Tighter Variational Bounds are Not Necessarily Bettercited by 76
  24. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  25. 2023Training Chain-of-Thought via Latent-Variable Inferencecited by 46
  26. 2025Reasoning to Learn from Latent Thoughtscited by 43
  27. 2023Joint Prompt Optimization of Stacked LLMs using Variational Inferencecited by 32
  28. 2020Making Sense of Reinforcement Learning and Probabilistic Inferencecited by 30
  29. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 27
  30. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 25
  31. 2020Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learningcited by 12
  32. 2024Averaging log-likelihoods in direct alignmentcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 18.

29 papers.

Sort:

By area:

  1. 2025Learning to Reason for Long-Form Story Generationcited by 34
  2. 2025Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Densecited by 19
  3. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 17
  4. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Taskscited by 16
  5. 2025Variational Reasoning for Language Modelscited by 8
  6. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 8
  7. 2025Escaping the Verifier: Learning to Reason via Demonstrationscited by 7
  8. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 6
  9. 2025The Imitation Game: Turing Machine Imitator is Length Generalizable Reasonercited by 3
  10. 2026Efficient Paths and Dense Rewards: Probabilistic Flow Reasoning for Large Language Modelscited by 2
  11. 2026Native Reasoning Models: Training Language Models to Reason on Unverifiable Datacited by 2
  12. 2026Scaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspacescited by 2
17 more
  1. 2026Self-Evolving Deep Research via Joint Generation and Evaluationcited by 2
  2. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 2
  3. 2025Coupled Variational Reinforcement Learning for Language Model General Reasoningcited by 1
  4. 2026DARL: Encouraging Diverse Answers for General Reasoning without Verifierscited by 1
  5. 2026REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judgecited by 1
  6. 2026AI Alignment via Incentives and Correctioncited by 1
  7. 2026Soft-SVeRL: Self-Verified Reinforcement Learning with Soft Rewardscited by 1
  8. 2026Intrinsic Selection and Particle Resampling for Inference-Time Scaling Beyond Domain Verifiabilitycited by 1
  9. 2026Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Driftcited by 1
  10. 2025Humanline: Online Alignment as Perceptual Losscited by 0
  11. 2025Bradley-Terry Policy Optimization for Generative Preference Modelingcited by 0
  12. 2025Learning to Reason in LLMs by Expectation Maximizationcited by 0
  13. 2026Trust Your Memory: Verifiable Control of Smart Homes through Reinforcement Learning with Multi-dimensional Rewardscited by 0
  14. 2026Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilitiescited by 0
  15. 2026Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Modelscited by 0
  16. 2026Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agentscited by 0
  17. 2026Rethinking Probability-Based Reinforcement Learning From Posterior Concentrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
  2. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  3. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning
  4. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
  5. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  6. 2026Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards
  7. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  8. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving