# Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies

arXiv: https://arxiv.org/abs/2111.02552 · HTML: https://mlsanity.com/paper/2111.02552/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 58 references that are in the map.
- First posted on arXiv 3 Nov 2021.

## Builds on (most cited first, 30 of 32)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Safe and Efficient Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1606.02647/) (2016)
- [Acme: A Research Framework for Distributed Reinforcement Learning](https://mlsanity.com/paper/2006.00979/) (2020)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [An empirical investigation of the challenges of real-world reinforcement learning](https://mlsanity.com/paper/2003.11881/) (2020)
- [Discretizing Continuous Action Space for On-Policy Optimization](https://mlsanity.com/paper/1901.10500/) (2019)
- [Remember and Forget for Experience Replay](https://mlsanity.com/paper/1807.05827/) (2018)
- [Value constrained model-free continuous control](https://mlsanity.com/paper/1902.04623/) (2019)
- [Relative Entropy Regularized Policy Iteration](https://mlsanity.com/paper/1812.02256/) (2018)
- [Growing Action Spaces](https://mlsanity.com/paper/1906.12266/) (2019)
- [Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning](https://mlsanity.com/paper/1903.08542/) (2019)
- [Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments](https://mlsanity.com/paper/1804.00361/) (2018)
- [Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking](https://mlsanity.com/paper/2011.07537/) (2020)
- [Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles](https://mlsanity.com/paper/2010.14641/) (2020)
- [Reinforcement Learning of Musculoskeletal Control from Functional Simulations](https://mlsanity.com/paper/2007.06669/) (2020)
- [MPC-based Reinforcement Learning for Economic Problems with Application to Battery Storage](https://mlsanity.com/paper/2104.02411/) (2021)
- [Discrete Action On-Policy Learning with Action-Value Critic](https://mlsanity.com/paper/2002.03534/) (2020)
- [High-Speed Finite Control Set Model Predictive Control for Power Electronics](https://mlsanity.com/paper/1510.05578/) (2015)
- [Decompositions and bang-bang properties](https://mlsanity.com/paper/1603.05362/) (2016)

## Built on by (most cited first, 30 of 38)

- [Solving Continuous Control via Q-learning](https://mlsanity.com/paper/2210.12566/) (2022)
- [Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance](https://mlsanity.com/paper/2206.03787/) (2022)
- [Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers](https://mlsanity.com/paper/2411.15370/) (2024)
- [Continuous Control with Coarse-to-fine Reinforcement Learning](https://mlsanity.com/paper/2407.07787/) (2024)
- [Reinforcement Learning with Simple Sequence Priors](https://mlsanity.com/paper/2305.17109/) (2023)
- [Bingham Policy Parameterization for 3D Rotations in Reinforcement Learning](https://mlsanity.com/paper/2202.03957/) (2022)
- [TC-Driver: Trajectory Conditioned Driving for Robust Autonomous Racing -- A Reinforcement Learning Approach](https://mlsanity.com/paper/2205.09370/) (2022)
- [Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution](https://mlsanity.com/paper/2404.04253/) (2024)
- [On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economics](https://mlsanity.com/paper/2310.19527/) (2023)
- [REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes](https://mlsanity.com/paper/2401.08850/) (2024)
- [Distributional Reinforcement Learning-based Energy Arbitrage Strategies in Imbalance Settlement Mechanism](https://mlsanity.com/paper/2401.00015/) (2023)
- [Stochastic Q-learning for Large Discrete Action Spaces](https://mlsanity.com/paper/2405.10310/) (2024)
- [Redundancy-aware Action Spaces for Robot Learning](https://mlsanity.com/paper/2406.04144/) (2024)
- [Using Simulation Optimization to Improve Zero-shot Policy Transfer of Quadrotors](https://mlsanity.com/paper/2201.01369/) (2022)
- [Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning](https://mlsanity.com/paper/2408.00309/) (2024)
- [Robust Path Following on Rivers Using Bootstrapped Reinforcement Learning](https://mlsanity.com/paper/2303.15178/) (2023)
- [Temporal-Difference Learning Using Distributed Error Signals](https://mlsanity.com/paper/2411.03604/) (2024)
- [An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces](https://mlsanity.com/paper/2411.11088/) (2024)
- [Model Predictive Control-Guided Reinforcement Learning for Implicit Balancing](https://mlsanity.com/paper/2510.04868/) (2025)
- [Sense, Imagine, Act: Multimodal Perception Improves Model-Based Reinforcement Learning for Head-to-Head Autonomous Racing](https://mlsanity.com/paper/2305.04750/) (2023)
- [A Statistical Guarantee for Representation Transfer in Multitask Imitation Learning](https://mlsanity.com/paper/2311.01589/) (2023)
- [Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies](https://mlsanity.com/paper/2404.18821/) (2024)
- [RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning](https://mlsanity.com/paper/2601.23075/) (2026)
- [Reward hacking in physical reinforcement learning revealed by turbulent drag reduction](https://mlsanity.com/paper/2606.06227/) (2026)
- [Position: Behavioral Systems Require Behavioral Tests](https://mlsanity.com/paper/2608.18081/) (2026)
- [GPU-Accelerated Policy Optimization via Batch Automatic Differentiation of Gaussian Processes for Real-World Control](https://mlsanity.com/paper/2202.13638/) (2022)
- [RangL: A Reinforcement Learning Competition Platform](https://mlsanity.com/paper/2208.00003/) (2022)
- [Game-theoretic Objective Space Planning](https://mlsanity.com/paper/2209.07758/) (2022)
- [Model-Free versus Model-Based Reinforcement Learning for Fixed-Wing UAV Attitude Control Under Varying Wind Conditions](https://mlsanity.com/paper/2409.17896/) (2024)
- [Learning in complex action spaces without policy gradients](https://mlsanity.com/paper/2410.06317/) (2024)

## Abstract (arXiv)

Reinforcement learning (RL) for continuous control typically employs distributions whose support covers the entire action space. In this work, we investigate the colloquially known phenomenon that trained agents often prefer actions at the boundaries of that space. We draw theoretical connections to the emergence of bang-bang behavior in optimal control, and provide extensive empirical evaluation across a variety of recent RL algorithms. We replace the normal Gaussian by a Bernoulli distribution that solely considers the extremes along each action dimension - a bang-bang controller. Surprisingly, this achieves state-of-the-art performance on several continuous control benchmarks - in contrast to robotic hardware, where energy and maintenance cost affect controller choices. Since exploration, learning,and the final solution are entangled in RL, we provide additional imitation learning experiments to reduce the impact of exploration on our analysis. Finally, we show that our observations generalize to environments that aim to model real-world challenges and evaluate factors to mitigate the emergence of bang-bang solutions. Our findings emphasize challenges for benchmarking continuous control algorithms, particularly in light of potential real-world applications.