mlsanityData to 7 Oct 2026

Value constrained model-free continuous control

arXiv 1902.04623 ↗First posted 12 Feb 2019cs.ROcited by 34

Not on the map. Why?

Abstract (arXiv)

The naive application of Reinforcement Learning algorithms to continuous control problems -- such as locomotion and manipulation -- often results in policies which rely on high-amplitude, high-frequency control signals, known colloquially as bang-bang control. Although such solutions may indeed maximize task reward, they can be unsuitable for real world systems. Bang-bang control may lead to increased wear and tear or energy consumption, and tends to excite undesired second-order dynamics. To counteract this issue, multi-objective optimization can be used to simultaneously optimize both the reward and some auxiliary cost that discourages undesired (e.g. high-amplitude) control. In principle, such an approach can yield the sought after, smooth, control policies. It can, however, be hard to find the correct trade-off between cost and return that results in the desired behavior. In this paper we propose a new constraint-based reinforcement learning approach that ensures task success while minimizing one or more auxiliary costs (such as control effort). We employ Lagrangian relaxation to learn both (a) the parameters of a control policy that satisfies the desired constraints and (b) the Lagrangian multipliers for the optimization. Moreover, we demonstrate that we can satisfy constraints either in expectation or in a per-step fashion, and can even learn a single policy that is able to dynamically trade-off between return and cost. We demonstrate the efficacy of our approach using a number of continuous control benchmark tasks, a realistic, energy-optimized quadruped locomotion task, as well as a reaching task on a real robot arm.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 12; 2023: 9; 2024: 5; 2025: 1; 2026: 1.

34 papers.

Sort:

By area:

  1. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  2. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  3. 2023State-wise Safe Reinforcement Learning: A Surveycited by 48
  4. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
  5. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 37
  6. 2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 28
  7. 2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 27
  8. 2022Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviorscited by 27
  9. 2022Safe Reinforcement Learning Using Black-Box Reachability Analysiscited by 22
  10. 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 18
  11. 2021Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translationcited by 17
  12. 2022Learning Soft Constraints From Constrained Expert Demonstrationscited by 13
22 more
  1. 2022Policy Optimization with Advantage Regularization for Long-Term Fairness in Decision Systemscited by 13
  2. 2023GUARD: A Safe Reinforcement Learning Benchmarkcited by 13
  3. 2023Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learningcited by 10
  4. 2026Machine Unlearning under Retain-Forget Entanglementcited by 10
  5. 2023State-wise Constrained Policy Optimizationcited by 9
  6. 2022The Real Deal: A Review of Challenges and Opportunities in Moving Reinforcement Learning-Based Traffic Signal Control Systems Towards Realitycited by 8
  7. 2024GLinSAT: The General Linear Satisfiability Neural Network Layer By Accelerated Gradient Descentcited by 7
  8. 2022Do You Need the Entropy Reward (in Practice)?cited by 6
  9. 2024Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolutioncited by 6
  10. 2023Reduced Policy Optimization for Continuous Control with Hard Constraintscited by 5
  11. 2022Conservative Distributional Reinforcement Learning with Safety Constraintscited by 3
  12. 2023Marketing Budget Allocation with Offline Constrained Deep Reinforcement Learningcited by 3
  13. 2023Approximate Shielding of Atari Agents for Safe Explorationcited by 2
  14. 2024Federated Reinforcement Learning with Constraint Heterogeneitycited by 2
  15. 2021Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific researchcited by 1
  16. 2022An Empirical Evaluation of Posterior Sampling for Constrained Reinforcement Learningcited by 1
  17. 2023Probabilistic Constrained Reinforcement Learning with Formal Interpretabilitycited by 1
  18. 2021Towards Autonomous Satellite Communications: An AI-based Framework to Address System-level Challengescited by 0
  19. 2023Reinforcement Learning in a Safety-Embedded MDP with Trajectory Optimizationcited by 0
  20. 2024Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfactioncited by 0
  21. 2024Effective Reward Specification in Deep Reinforcement Learningcited by 0
  22. 2025Multi-Agent Reinforcement Learning for Task Offloading in Wireless Edge Networkscited by 0