Unthrottling the Tanh Jacobian in SAC: A Negative Result on Bang-Bang Control and MetaDrive
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
Soft Actor-Critic (SAC) represents a continuous policy as an unbounded Gaussian that is squashed by tanh. The Jacobian of that map is partial a/partial u = 1-a2, which vanishes as |a|→ 1. A natural concern is that this throttle starves the actor of critic signal exactly where extreme actions (full brake, full throttle) are optimal. We test a minimal intervention that restores the missing signal: one extra term in the actor loss whose gradient on the pre-tanh mean is the detached action-gradient of Q, with no gain parameter. On a minimum-time double integrator whose optimum is bang-bang at the action bounds, vanilla SAC already reaches near-optimal return (-31.6 vs. a calibrated optimum of -30.3) across ten paired seeds. An ungated bypass does saturate the policy (99% of eval steps with |a|≥ 0.9) and collapses return to -195.5. A gated bypass that fires only on the flat shoulder |a|∈[0.9,0.999] also fails, and does so without leaving a saturated policy. Warm-started MetaDrive fine-tuning shows the same pattern: the bypass does not improve return, and where collision rate falls it is typically traded for out-of-road departures. Auto-tuned entropy coefficient rises against the bypass, which is a push toward the tails. The Jacobian effect is real. Treating it as a bug to be undone is not free, and on the tasks studied here it is not helpful. Saturating a bound is not the same as solving a problem whose optimum lives on that bound.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 3 of its 7 references that are in the map. First posted on arXiv 8 Sep 2026.
Builds on
3 of 7 references.
References · 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns
- 2026An analysis of Mirror-Descent Soft Actor-Critic
- 2025Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulator
- 2026Refined Analysis of Entropy-Regularized Actor-Critic
- 2024Rethinking Soft Actor-Critic in High-Dimensional Action Spaces: The Cost of Ignoring Distribution Shift
- 2026DAMPER: Return-Prioritized Gradient Control for Smooth Policies
- 2026Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods
- 2024Compatible Gradient Approximations for Actor-Critic Algorithms