mlsanityData to 7 Oct 2026

SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy

arXiv 2609.12749 ↗First posted 11 Sep 2026cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline-to-online reinforcement learning reduces interaction cost for real-world robot learning but suffers from persistent value estimation instability. Existing methods address this through pessimistic regularization, lower-bound calibration, and architectural normalization, but an overlooked source of instability lies in the entropy formulation: the standard log-entropy term can become negative, destabilizing policy updates. We introduce SCQ (Sigmoid-Bounded Conservative Q-Learning), which replaces this term with a sigmoid-bounded formulation that stays strictly positive. SCQ retains conservative Q regularization and return-based lower-bound calibration, stabilizing policy optimization without sacrificing exploration. We evaluate SCQ on D4RL (Minari) benchmarks under both single-demonstration and standard dataset settings, as well as on simulation and real-world visual tasks. SCQ matches or exceeds baseline performance while exhibiting more stable training dynamics across state-based and visual benchmarks, and transfers to four real-robot platforms including manipulation, wheeled, quadruped, and humanoid systems. A direct clipping intervention that removes negative log-probability contributions, together with gradient-matched positive-score controls, indicates that positivity rather than a particular score shape alone drives much of the improvement. Project website: https://scq-rl.github.io.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 18 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

17 of 18 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun

Other references · 13

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  3. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  4. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  5. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  6. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 81
  7. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  8. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  9. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
  10. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  11. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 7
  12. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
1 more
  1. 2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RLcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Off-Policy Actor-Critic with Sigmoid-Bounded Entropy for Real-World Robot Learning
  2. 2024Strategically Conservative Q-Learning
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  4. 2026RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking
  5. 2025EXPO: Stable Reinforcement Learning with Expressive Policies
  6. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
  7. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds
  8. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning