# Progress Constraints for Reinforcement Learning in Behavior Trees

arXiv: https://arxiv.org/abs/2602.06525 · HTML: https://mlsanity.com/paper/2602.06525/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 23 of 23)

- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [A Closer Look at Invalid Action Masking in Policy Gradient Algorithms](https://mlsanity.com/paper/2006.14171/) (2020)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [Learning to Walk in the Real World with Minimal Human Effort](https://mlsanity.com/paper/2002.08550/) (2020)
- [Behavior Trees in Robotics and AI: An Introduction](https://mlsanity.com/paper/1709.00084/) (2017)
- [The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems](https://mlsanity.com/paper/2309.05837/) (2023)
- [Reachability Constrained Reinforcement Learning](https://mlsanity.com/paper/2205.07536/) (2022)
- [A Survey of Behavior Trees in Robotics and AI](https://mlsanity.com/paper/2005.05842/) (2020)
- [Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)](https://mlsanity.com/paper/2112.15422/) (2021)
- [Towards Blended Reactive Planning and Acting using Behavior Trees](https://mlsanity.com/paper/1611.00230/) (2016)
- [Behavior Trees in Robot Control Systems](https://mlsanity.com/paper/2203.13083/) (2022)
- [On modularity in reactive control architectures, with an application to formal verification](https://mlsanity.com/paper/2008.12515/) (2020)
- [A Framework for Constrained and Adaptive Behavior-Based Agents](https://mlsanity.com/paper/1506.02312/) (2015)
- [Multi-Advisor Reinforcement Learning](https://mlsanity.com/paper/1704.00756/) (2017)
- [Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning](https://mlsanity.com/paper/2310.02360/) (2023)
- [Improving the Performance of Backward Chained Behavior Trees that use Reinforcement Learning](https://mlsanity.com/paper/2112.13744/) (2021)
- [Improving the performance of Learned Controllers in Behavior Trees using Value Function Estimates at Switching Boundaries](https://mlsanity.com/paper/2305.18903/) (2023)

## Abstract (arXiv)

Behavior Trees (BTs) provide a structured and reactive framework for decision-making, commonly used to switch between sub-controllers based on environmental conditions. Reinforcement Learning (RL), on the other hand, can learn near-optimal controllers but sometimes struggles with sparse rewards, safe exploration, and long-horizon credit assignment. Combining BTs with RL has the potential for mutual benefit: a BT design encodes structured domain knowledge that can simplify RL training, while RL enables automatic learning of the controllers within BTs. However, naive integration of BTs and RL can lead to some controllers counteracting other controllers, possibly undoing previously achieved subgoals, thereby degrading the overall performance. To address this, we propose progress constraints, a novel mechanism where feasibility estimators constrain the allowed action set based on theoretical BT convergence results. Empirical evaluations in a 2D proof-of-concept and a high-fidelity warehouse environment demonstrate improved performance, sample efficiency, and constraint satisfaction, compared to prior methods of BT-RL integration.