mlsanityData to 14 Oct 2026

LLQL: Logistic Likelihood Q-Learning for Reinforcement Learning

arXiv 2307.02345 ↗First posted 5 Jul 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern reinforcement learning (RL) can be categorized into online and offline variants. As a pivotal aspect of both online and offline RL, current research on the Bellman equation revolves primarily around optimization techniques and performance enhancement rather than exploring the inherent structural properties of the Bellman error, such as its distribution characteristics. This study investigates the distribution of the Bellman approximation error through iterative exploration of the Bellman equation with the observation that the Bellman error approximately follows the Logistic distribution. Based on this, we proposed the utilization of the Logistic maximum likelihood function (LLoss) as an alternative to the commonly used mean squared error (MSELoss) that assumes a Normal distribution for Bellman errors. We validated the hypotheses through extensive numerical experiments across diverse online and offline environments. In particular, we applied the Logistic correction to loss functions in various RL baseline methods and observed that the results with LLoss consistently outperformed the MSE counterparts. We also conducted the Kolmogorov-Smirnov tests to confirm the reliability of the Logistic distribution. Moreover, our theory connects the Bellman error to the proportional reward scaling phenomenon by providing a distribution-based analysis. Furthermore, we applied the bias-variance decomposition for sampling from the Logistic distribution. The theoretical and empirical insights of this study lay a valuable foundation for future investigations and enhancements centered on the distribution of Bellman error.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 46 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

25 of 46 references.

References · 25

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2019Model-Based Reinforcement Learning for Ataricited by 502
  9. 2020POMO: Policy Optimization with Multiple Optima for Reinforcement Learningcited by 348
  10. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  11. 2017A unified view of entropy-regularized Markov decision processescited by 182
  12. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
13 more
  1. 2019Behaviour Suite for Reinforcement Learningcited by 132
  2. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  4. 2021Efficient Active Search for Combinatorial Optimization Problemscited by 100
  5. 2019Deep Reinforcement Learning for Tradingcited by 73
  6. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  7. 2022Learning Generalizable Models for Vehicle Routing Problems via Knowledge Distillationcited by 62
  8. 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 38
  9. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  10. 2020Logistic Q-Learningcited by 34
  11. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 32
  12. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  13. 2022Reinforcement learning for automatic quadrilateral mesh generation: a soft actor-critic approachcited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ShiQ: Bringing back Bellman to LLMs
  2. 2024The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximation
  3. 2024Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
  4. 2025Bellman Calibration for V-Learning in Offline Reinforcement Learning
  5. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  6. 2024UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learning
  7. 2023Revisiting Bellman Errors for Offline Model Selection
  8. 2025Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learning