# Boosting Soft Q-Learning by Bounding

arXiv: https://arxiv.org/abs/2406.18033 · HTML: https://mlsanity.com/paper/2406.18033/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 26 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 17 of 17)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/2007.04938/) (2020)
- [Deep Hierarchical Planning from Pixels](https://mlsanity.com/paper/2206.04114/) (2022)
- [Composable Deep Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/1803.06773/) (2018)
- [Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress](https://mlsanity.com/paper/2206.01626/) (2022)
- [Identifiability in inverse reinforcement learning](https://mlsanity.com/paper/2106.03498/) (2021)
- [Global optimization of Lipschitz functions](https://mlsanity.com/paper/1703.02628/) (2017)
- [Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening](https://mlsanity.com/paper/1611.01606/) (2016)
- [Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates](https://mlsanity.com/paper/2112.15025/) (2021)
- [Self-Imitation Learning via Generalized Lower Bound Q-learning](https://mlsanity.com/paper/2006.07442/) (2020)
- [Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning](https://mlsanity.com/paper/2212.01174/) (2022)
- [Qgraph-bounded Q-learning: Stabilizing Model-Free Off-Policy Deep Reinforcement Learning](https://mlsanity.com/paper/2007.07582/) (2020)
- [Bounding the Optimal Value Function in Compositional Reinforcement Learning](https://mlsanity.com/paper/2303.02557/) (2023)

## Built on by (most cited first, 4 of 4)

- [Bootstrapped Reward Shaping](https://mlsanity.com/paper/2501.00989/) (2025)
- [Causality-informed Anomaly Detection in Partially Observable Sensor Networks: Moving beyond Correlations](https://mlsanity.com/paper/2507.09742/) (2025)
- [Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examples](https://mlsanity.com/paper/2407.03311/) (2024)
- [Provably Efficient Reward Transfer in Reinforcement Learning with Discrete Markov Decision Processes](https://mlsanity.com/paper/2503.13414/) (2025)

## Abstract (arXiv)

An agent's ability to leverage past experience is critical for efficiently solving new tasks. Prior work has focused on using value function estimates to obtain zero-shot approximations for solutions to a new task. In soft Q-learning, we show how any value function estimate can also be used to derive double-sided bounds on the optimal value function. The derived bounds lead to new approaches for boosting training performance which we validate experimentally. Notably, we find that the proposed framework suggests an alternative method for updating the Q-function, leading to boosted performance.