# Qualitative Measurements of Policy Discrepancy for Return-Based Deep Q-Network

arXiv: https://arxiv.org/abs/1806.06953 · HTML: https://mlsanity.com/paper/1806.06953/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2018.

## Built on by (most cited first, 4 of 4)

- [The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure](https://mlsanity.com/paper/2205.10047/) (2022)
- [Quality Diversity Evolutionary Learning of Decision Trees](https://mlsanity.com/paper/2208.12758/) (2022)
- [Continuous Deep Q-Learning with Simulator for Stabilization of Uncertain Discrete-Time Systems](https://mlsanity.com/paper/2101.05640/) (2021)
- [Optimize Neural Fictitious Self-Play in Regret Minimization Thinking](https://mlsanity.com/paper/2104.10845/) (2021)

## Abstract (arXiv)

The deep Q-network (DQN) and return-based reinforcement learning are two promising algorithms proposed in recent years. DQN brings advances to complex sequential decision problems, while return-based algorithms have advantages in making use of sample trajectories. In this paper, we propose a general framework to combine DQN and most of the return-based reinforcement learning algorithms, named R-DQN. We show the performance of traditional DQN can be improved effectively by introducing return-based reinforcement learning. In order to further improve the R-DQN, we design a strategy with two measurements which can qualitatively measure the policy discrepancy. Moreover, we give the two measurements' bounds in the proposed R-DQN framework. We show that algorithms with our strategy can accurately express the trace coefficient and achieve a better approximation to return. The experiments, conducted on several representative tasks from the OpenAI Gym library, validate the effectiveness of the proposed measurements. The results also show that the algorithms with our strategy outperform the state-of-the-art methods.