# RAVR: Reference-Answer-guided Variational Reasoning for Large Language Models

arXiv: https://arxiv.org/abs/2510.25206 · HTML: https://mlsanity.com/paper/2510.25206/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 23 of 23)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning](https://mlsanity.com/paper/2504.11456/) (2025)
- [REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization](https://mlsanity.com/paper/2501.03262/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [Thought Anchors: Which LLM Reasoning Steps Matter?](https://mlsanity.com/paper/2506.19143/) (2025)
- [RLPR: Extrapolating RLVR to General Domains without Verifiers](https://mlsanity.com/paper/2506.18254/) (2025)
- [Reinforcing General Reasoning without Verifiers](https://mlsanity.com/paper/2505.21493/) (2025)
- [Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiency](https://mlsanity.com/paper/2506.08343/) (2025)
- [Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning](https://mlsanity.com/paper/2508.08221/) (2025)
- [StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason](https://mlsanity.com/paper/2507.02841/) (2025)
- [Reverse-Engineered Reasoning for Open-Ended Generation](https://mlsanity.com/paper/2509.06160/) (2025)
- [Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning](https://mlsanity.com/paper/2504.13941/) (2025)
- [Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Tasks](https://mlsanity.com/paper/2506.13351/) (2025)
- [ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning](https://mlsanity.com/paper/2507.02834/) (2025)
- [Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation](https://mlsanity.com/paper/2506.15068/) (2025)
- [Reinforcement Learning Teachers of Test Time Scaling](https://mlsanity.com/paper/2506.08388/) (2025)

## Built on by (most cited first, 8 of 8)

- [Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning](https://mlsanity.com/paper/2605.02913/) (2026)
- [Coupled Variational Reinforcement Learning for Language Model General Reasoning](https://mlsanity.com/paper/2512.12576/) (2025)
- [Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learning](https://mlsanity.com/paper/2607.00461/) (2026)
- [Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization](https://mlsanity.com/paper/2607.14614/) (2026)
- [Efficient LLM Reasoning via Variational Posterior Guidance with Efficiency Awareness](https://mlsanity.com/paper/2605.11019/) (2026)
- [A First-Principles Theory of Slow Thinking and Active Perception](https://mlsanity.com/paper/2607.08196/) (2026)
- [Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference](https://mlsanity.com/paper/2607.08724/) (2026)
- [Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data](https://mlsanity.com/paper/2610.18878/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) can refine the reasoning abilities of large language models (LLMs), but critically depends on a key prerequisite: the LLM can already generate high-utility reasoning paths with non-negligible probability. For tasks beyond the LLM's current competence, such reasoning path can be hard to sample, and learning risks reinforcing familiar but suboptimal reasoning. We are motivated by the insight from cognitive science that Why is this the answer is often an easier question than What is the answer, as it avoids the heavy cognitive load of open-ended exploration, opting instead for explanatory reconstruction-systematically retracing the reasoning that links a question to its answer. We show that LLMs can similarly leverage answers to derive high-quality reasoning paths. We formalize this phenomenon and prove that conditioning on answer provably increases the expected utility of sampled reasoning paths, thereby transforming intractable problems into learnable ones. Building on this insight, we introduce RAVR (Reference-Answer-guided Variational Reasoning), an end-to-end framework that uses answer-conditioned reasoning as a variational surrogate for question-only reasoning. Experiments in both general and math domains demonstrate consistent improvements over strong baselines. We further analyze the reasoning behavior and find that RAVR reduces hesitation, strengthens conclusion consolidation, and promotes problem-specific strategies in reasoning.