# Continuous-Discrete Reinforcement Learning for Hybrid Control in Robotics

arXiv: https://arxiv.org/abs/2001.00449 · HTML: https://mlsanity.com/paper/2001.00449/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2020.

## Built on by (most cited first, 30 of 44)

- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [Residual Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/2106.08050/) (2021)
- [HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation](https://mlsanity.com/paper/2305.03942/) (2023)
- [Solving Continuous Control via Q-learning](https://mlsanity.com/paper/2210.12566/) (2022)
- [A Survey on Physics Informed Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2309.01909/) (2023)
- [Continuous Control with Action Quantization from Demonstrations](https://mlsanity.com/paper/2110.10149/) (2021)
- [Robot Learning of Mobile Manipulation with Reachability Behavior Priors](https://mlsanity.com/paper/2203.04051/) (2022)
- [Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking](https://mlsanity.com/paper/2406.03704/) (2024)
- [TAAC: Temporally Abstract Actor-Critic for Continuous Control](https://mlsanity.com/paper/2104.06521/) (2021)
- [Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2101.09458/) (2021)
- [Multi-user Co-inference with Batch Processing Capable Edge Server](https://mlsanity.com/paper/2206.06304/) (2022)
- [Automated Synthesis of Steady-State Continuous Processes using Reinforcement Learning](https://mlsanity.com/paper/2101.04422/) (2021)
- [Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning](https://mlsanity.com/paper/2201.09765/) (2022)
- [Monte Carlo Tree Search based Hybrid Optimization of Variational Quantum Circuits](https://mlsanity.com/paper/2203.16707/) (2022)
- [MO2: Model-Based Offline Options](https://mlsanity.com/paper/2209.01947/) (2022)
- [Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience](https://mlsanity.com/paper/2109.11767/) (2021)
- [Model-based Reinforcement Learning for Parameterized Action Spaces](https://mlsanity.com/paper/2404.03037/) (2024)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution](https://mlsanity.com/paper/2404.04253/) (2024)
- [A Hybrid Approach for Learning to Shift and Grasp with Elaborate Motion Primitives](https://mlsanity.com/paper/2111.01510/) (2021)
- [SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration](https://mlsanity.com/paper/2211.13743/) (2022)
- [Discretionary Lane-Change Decision and Control via Parameterized Soft Actor-Critic for Hybrid Action Space](https://mlsanity.com/paper/2402.15790/) (2024)
- [MetaGFN: Exploring Distant Modes with Adapted Metadynamics for Continuous GFlowNets](https://mlsanity.com/paper/2408.15905/) (2024)
- [One model Packs Thousands of Items with Recurrent Conditional Query Learning](https://mlsanity.com/paper/2111.06726/) (2021)
- [Dynamic Neighborhood Construction for Structured Large Discrete Action Spaces](https://mlsanity.com/paper/2305.19891/) (2023)
- [Boosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel States](https://mlsanity.com/paper/2210.00211/) (2022)
- [Driver Assistance Eco-driving and Transmission Control with Deep Reinforcement Learning](https://mlsanity.com/paper/2212.07594/) (2022)
- [Residual Policy Learning for Powertrain Control](https://mlsanity.com/paper/2212.07611/) (2022)
- [Optimal control barrier functions for RL based safe powertrain control](https://mlsanity.com/paper/2405.11391/) (2024)

## Abstract (arXiv)

Many real-world control problems involve both discrete decision variables - such as the choice of control modes, gear switching or digital outputs - as well as continuous decision variables - such as velocity setpoints, control gains or analogue outputs. However, when defining the corresponding optimal control or reinforcement learning problem, it is commonly approximated with fully continuous or fully discrete action spaces. These simplifications aim at tailoring the problem to a particular algorithm or solver which may only support one type of action space. Alternatively, expert heuristics are used to remove discrete actions from an otherwise continuous space. In contrast, we propose to treat hybrid problems in their 'native' form by solving them with hybrid reinforcement learning, which optimizes for discrete and continuous actions simultaneously. In our experiments, we first demonstrate that the proposed approach efficiently solves such natively hybrid reinforcement learning problems. We then show, both in simulation and on robotic hardware, the benefits of removing possibly imperfect expert-designed heuristics. Lastly, hybrid reinforcement learning encourages us to rethink problem definitions. We propose reformulating control problems, e.g. by adding meta actions, to improve exploration or reduce mechanical wear and tear.