# Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control

arXiv: https://arxiv.org/abs/2604.03023 · HTML: https://mlsanity.com/paper/2604.03023/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 3 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 20 of 20)

- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [Autonomous Vehicles on the Edge: A Survey on Autonomous Vehicle Racing](https://mlsanity.com/paper/2202.07008/) (2022)
- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [Auxiliary Tasks Speed Up Learning PointGoal Navigation](https://mlsanity.com/paper/2007.04561/) (2020)
- [Bridging State and History Representations: Understanding Self-Predictive RL](https://mlsanity.com/paper/2401.08898/) (2024)
- [Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey](https://mlsanity.com/paper/1909.02964/) (2019)
- [Modeling Human Driving Behavior through Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2006.06412/) (2020)
- [Understanding Self-Predictive Learning for Reinforcement Learning](https://mlsanity.com/paper/2212.03319/) (2022)
- [MEGA-DAgger: Imitation Learning with Multiple Imperfect Experts](https://mlsanity.com/paper/2303.00638/) (2023)
- [When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learning](https://mlsanity.com/paper/2406.17718/) (2024)
- [A Benchmark Comparison of Imitation Learning-based Control Policies for Autonomous Racing](https://mlsanity.com/paper/2209.15073/) (2022)
- [BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human Racing Gameplay](https://mlsanity.com/paper/2402.14194/) (2024)
- [An Adaptive Human Driver Model for Realistic Race Car Simulations](https://mlsanity.com/paper/2203.01909/) (2022)

## Abstract (arXiv)

Learning high-performance control policies that remain consistent with expert behavior is a fundamental challenge in robotics. Reinforcement learning can discover high-performing strategies but often departs from desirable human behavior, whereas imitation learning is limited by demonstration quality and struggles to improve beyond expert data. We propose a behavior-constrained reinforcement learning framework that improves beyond demonstrations while explicitly controlling deviation from expert behavior. Because expert-consistent behavior in dynamic control is inherently trajectory-level, we introduce a receding-horizon predictive mechanism that models short-term future trajectories and provides look-ahead rewards during training. To account for the natural variability of human behavior under disturbances and changing conditions, we further condition the policy on reference trajectories, allowing it to represent a distribution of expert-consistent behaviors rather than a single deterministic target. Empirically, we evaluate the approach in high-fidelity race car simulation using data from professional drivers, a domain characterized by extreme dynamics and narrow performance margins. The learned policies achieve competitive lap times while maintaining close alignment with expert driving behavior, outperforming baseline methods in both performance and imitation quality. Beyond standard benchmarks, we conduct human-grounded evaluation in a driver-in-the-loop simulator and show that the learned policies reproduce setup-dependent driving characteristics consistent with the feedback of top-class professional race drivers. These results demonstrate that our method enables learning high-performance control policies that are both optimal and behavior-consistent, and can serve as reliable surrogates for human decision-making in complex control systems.