Data to 7 Oct 2026

Bridging the Gap Between Value and Policy Based Reinforcement Learning

arXiv 1702.08892 ↗First posted 28 Feb 2017cs.AI cs.LG stat.MLcited by 178

Not on the map. Why?

Abstract (arXiv)

We establish a new connection between value and policy based reinforcement learning (RL) based on a relationship between softmax temporal value consistency and policy optimality under entropy regularization. Specifically, we show that softmax consistent action values correspond to optimal entropy regularized policy probabilities along any action sequence, regardless of provenance. From this observation, we develop a new RL algorithm, Path Consistency Learning (PCL), that minimizes a notion of soft consistency error along multi-step action sequences extracted from both on- and off-policy traces. We examine the behavior of PCL in different scenarios and show that PCL can be interpreted as generalizing both actor-critic and Q-learning algorithms. We subsequently deepen the relationship by showing how a single model can be used to represent both a policy and the corresponding softmax state values, eliminating the need for a separate critic. The experimental evaluation demonstrates that PCL significantly outperforms strong actor-critic and Q-learning baselines across several benchmarks.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 2; 2020: 2; 2021: 31; 2022: 28; 2023: 32; 2024: 23; 2025: 39; 2026: 20.

178 papers.

Sort:

By area:

2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
2021GFlowNet Foundationscited by 286
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
2025Reasoning with Exploration: An Entropy Perspectivecited by 277
2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
2022Biological Sequence Design with GFlowNetscited by 172
2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
2018Self-Imitation Learningcited by 135

38 more

2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 129
2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
2023Amortizing intractable inference in large language modelscited by 94
2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
2021A general sample complexity analysis of vanilla policy gradientcited by 74
2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
2025Efficient Online Reinforcement Learning for Diffusion Policycited by 61
2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guaranteescited by 42
2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 40
2024Genetic-guided GFlowNets for Sample Efficient Molecular Optimizationcited by 36
2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 33
2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
2023Thompson sampling for improved exploration in GFlowNetscited by 31
2023Maximum entropy GFlowNets with soft Q-learningcited by 30
2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
2023Decision S4: Efficient Sequence-Based RL via State Spaces Layerscited by 26
2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
2021A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learningcited by 24
2025Generative Auto-Bidding with Value-Guided Explorationscited by 23
2022Your Policy Regularizer is Secretly an Adversarycited by 21
2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 20
2022Independent and Decentralized Learning in Markov Potential Gamescited by 19
2023Maximum Entropy Heterogeneous-Agent Reinforcement Learningcited by 19
2025Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Modelscited by 19
2022Regularized Gradient Descent Ascent for Two-Player Zero-Sum Markov Gamescited by 18

50 most cited of 178 All 178 (JSON)