mlsanityData to 14 Oct 2026

Frictional Q-Learning

arXiv 2509.19771 ↗First posted 24 Sep 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Off-policy reinforcement learning suffers from extrapolation errors when a learned policy selects actions that are weakly supported in the replay buffer. In this study, we address this issue by drawing an analogy to static friction. From this perspective, the replay buffer is represented as a smooth, low-dimensional action manifold, where the support directions correspond to the tangential component, while the normal component captures the dominant first-order extrapolation error. This decomposition reveals an intrinsic anisotropy in value sensitivity that naturally induces a stability condition analogous to a friction threshold. To mitigate deviations toward unsupported actions, we propose Frictional Q-Learning, an off-policy algorithm that encodes supported actions as tangent directions using a contrastive variational autoencoder. We further show that an orthonormal basis of the orthogonal complement corresponds to normal components under mild local isometry assumptions. Extensive empirical results on standard continuous-control benchmarks consistently demonstrate robust and stable performance compared with competitive baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 65 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

30 of 65 references.

References · 30

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  9. 2018DeepMind Control Suitecited by 956
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  12. 2018Disentangling by Factorisingcited by 616
18 more
  1. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  2. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  3. 2018Selective Experience Replay for Lifelong Learningcited by 241
  4. 2018Sorting out Lipschitz function approximationcited by 213
  5. 2020Revisiting Fundamentals of Experience Replaycited by 175
  6. 2018The Mechanics of n-Player Differentiable Gamescited by 118
  7. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 91
  8. 2022q-Learning in Continuous Timecited by 91
  9. 2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80
  10. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  11. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 74
  12. 2017Stein Variational Policy Gradientcited by 44
  13. 2020Stochastic Hamiltonian Gradient Methods for Smooth Gamescited by 32
  14. 2019Contrastive Variational Autoencoder Enhances Salient Featurescited by 28
  15. 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flowcited by 26
  16. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Ratecited by 10
  17. 2022Action Candidate Driven Clipped Double Q-learning for Discrete and Continuous Action Taskscited by 9
  18. 2021Interpretable performance analysis towards offline reinforcement learning: A dataset perspectivecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
  2. 2026From Policy Evaluation to Off-Policy Control: Provable Emergence of In-Context Q-Learning
  3. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  4. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  5. 2023Contextual Conservative Q-Learning for Offline Reinforcement Learning
  6. 2026Commit to the Bit: Reactive Reinforcement Learning Done Right
  7. 2026Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods
  8. 2026Q-learning Penalized Transformer for Safe Offline Reinforcement Learning