# Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving

arXiv: https://arxiv.org/abs/2610.09763 · HTML: https://mlsanity.com/paper/2610.09763/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving.
- Builds on 16 of its 43 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Placed in

- End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/)
- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 16 of 16)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Diffusion-Based Planning for Autonomous Driving with Flexible Guidance](https://mlsanity.com/paper/2501.15564/) (2025)
- [Baidu Apollo EM Motion Planner](https://mlsanity.com/paper/1807.08048/) (2018)
- [PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Driving](https://mlsanity.com/paper/2404.14327/) (2024)
- [CaRL: Learning Scalable Planning Policies with Simple Rewards](https://mlsanity.com/paper/2504.17838/) (2025)
- [CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning](https://mlsanity.com/paper/2403.19918/) (2024)
- [UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning](https://mlsanity.com/paper/2111.11097/) (2021)
- [Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning](https://mlsanity.com/paper/2502.04778/) (2025)
- [Uncertainty-Aware Decision Transformer for Stochastic Driving Environments](https://mlsanity.com/paper/2309.16397/) (2023)
- [iPLAN: Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.06236/) (2023)
- [Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning](https://mlsanity.com/paper/2609.06882/) (2026)

## Abstract (arXiv)

Offline reinforcement learning enables reward-driven policy improvement from fixed datasets without requiring online exploration, making it particularly attractive in safety-critical domains. A central challenge, however, is distribution shift: policy optimization may favor actions that are weakly supported by the offline data, rendering value estimates unreliable. Existing approaches primarily control this shift in the policy's own action space. In interactive environments such as autonomous driving, this can be insufficient: a candidate ego trajectory may remain well supported under the marginal behavior distribution while being poorly supported jointly with the surrounding-agent behavior observed in the logged interaction. We refer to this degradation in interaction support as interaction distribution shift (IDS), and introduce Interaction-Constrained Drive Policy (ICDP), an offline reinforcement learning framework that explicitly controls interaction-level distribution shift. Starting from the joint data distribution over ego and surrounding-agent futures, we show that joint-support degradation decomposes exactly into an ego-support component and a residual interaction-support component. We recover the latter through contrastive density-ratio estimation, isolating interaction compatibility without explicit joint-density modeling, surrounding-agent prediction, or rollouts in reactive simulators or learned world models during policy optimization. Closed-loop evaluations on nuPlan, Interplan and real-world truck experiments show that ICDP suppresses high-value yet interaction-unsupported trajectory selections and improves performance in interaction-critical driving scenarios. Project webpage: https://mahmoud-selim.github.io/ICDP/