# Bilevel Multi-Armed Bandit-Based Hierarchical Reinforcement Learning for Interaction-Aware Self-Driving at Unsignalized Intersections

arXiv: https://arxiv.org/abs/2502.03960 · HTML: https://mlsanity.com/paper/2502.03960/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 15 of its 41 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 15 of 15)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learning](https://mlsanity.com/paper/2103.14666/) (2021)
- [Policy Search for Model Predictive Control with Application to Agile Drone Flight](https://mlsanity.com/paper/2112.03850/) (2021)
- [State Dropout-Based Curriculum Reinforcement Learning for Self-Driving at Unsignalized Intersections](https://mlsanity.com/paper/2207.04361/) (2022)
- [Behavior Planning at Urban Intersections through Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2011.04697/) (2020)
- [Decentralized iLQR for Cooperative Trajectory Planning of Connected Autonomous Vehicles via Dual Consensus ADMM](https://mlsanity.com/paper/2301.04386/) (2023)
- [Reward-Driven Automated Curriculum Learning for Interaction-Aware Self-Driving at Unsignalized Intersections](https://mlsanity.com/paper/2403.13674/) (2024)
- [Learning When to Drive in Intersections by Combining Reinforcement Learning and Model Predictive Control](https://mlsanity.com/paper/1908.00177/) (2019)
- [A Two-Stage Optimization-based Motion Planner for Safe Urban Driving](https://mlsanity.com/paper/2002.02215/) (2020)
- [Game-theoretic Modeling of Traffic in Unsignalized Intersection Network for Autonomous Vehicle Control Verification and Validation](https://mlsanity.com/paper/1910.07141/) (2019)
- [Thompson Sampling for Bandits with Clustered Arms](https://mlsanity.com/paper/2109.01656/) (2021)
- [Chance-Aware Lane Change with High-Level Model Predictive Control Through Curriculum Reinforcement Learning](https://mlsanity.com/paper/2303.03723/) (2023)
- [Spatiotemporal Receding Horizon Control with Proactive Interaction Towards Autonomous Driving in Dense Traffic](https://mlsanity.com/paper/2308.05929/) (2023)

## Built on by (most cited first, 4 of 4)

- [SEG-Parking: Towards Safe, Efficient, and Generalizable Autonomous Parking via End-to-End Offline Reinforcement Learning](https://mlsanity.com/paper/2509.13956/) (2025)
- [Orchestrate, Generate, Reflect: A VLM-Based Multi-Agent Collaboration Framework for Automated Driving Policy Learning](https://mlsanity.com/paper/2509.17042/) (2025)
- [SocialTraj: Two-Stage Socially-Aware Trajectory Prediction for Autonomous Driving via Conditional Diffusion Model](https://mlsanity.com/paper/2509.17850/) (2025)
- [LUNA-AD: Lightweight Uncertainty-Aware Language Model with Lifelong Learning for Autonomous Driving](https://mlsanity.com/paper/2606.08470/) (2026)

## Abstract (arXiv)

In this work, we present BiM-ACPPO, a bilevel multi-armed bandit-based hierarchical reinforcement learning framework for interaction-aware decision-making and planning at unsignalized intersections. Essentially, it proactively takes the uncertainties associated with surrounding vehicles (SVs) into consideration, which encompass those stemming from the driver's intention, interactive behaviors, and the varying number of SVs. Intermediate decision variables are introduced to enable the high-level RL policy to provide an interaction-aware reference, for guiding low-level model predictive control (MPC) and further enhancing the generalization ability of the proposed framework. By leveraging the structured nature of self-driving at unsignalized intersections, the training problem of the RL policy is modeled as a bilevel curriculum learning task, which is addressed by the proposed Exp3.S-based BiMAB algorithm. It is noteworthy that the training curricula are dynamically adjusted, thereby facilitating the sample efficiency of the RL training process. Comparative experiments are conducted in the high-fidelity CARLA simulator, and the results indicate that our approach achieves superior performance compared to all baseline methods. Furthermore, experimental results in two new urban driving scenarios clearly demonstrate the commendable generalization performance of the proposed method.