# Formulation of Deep Reinforcement Learning Architecture Toward Autonomous Driving for On-Ramp Merge

arXiv: https://arxiv.org/abs/1709.02066 · HTML: https://mlsanity.com/paper/1709.02066/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2017.

## Built on by (most cited first, 19 of 19)

- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving](https://mlsanity.com/paper/2405.01440/) (2024)
- [TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learning](https://mlsanity.com/paper/2502.01387/) (2025)
- [A Multi-Agent Deep Reinforcement Learning Coordination Framework for Connected and Automated Vehicles at Merging Roadways](https://mlsanity.com/paper/2109.11672/) (2021)
- [Increasing the Efficiency of Policy Learning for Autonomous Vehicles by Multi-Task Representation Learning](https://mlsanity.com/paper/2103.14718/) (2021)
- [Encoding Distributional Soft Actor-Critic for Autonomous Driving in Multi-lane Scenarios](https://mlsanity.com/paper/2109.05540/) (2021)
- [Towards Learning Generalizable Driving Policies from Restricted Latent Representations](https://mlsanity.com/paper/2111.03688/) (2021)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [Safe, Efficient, Comfort, and Energy-saving Automated Driving through Roundabout Based on Deep Reinforcement Learning](https://mlsanity.com/paper/2306.11465/) (2023)
- [Comprehensive Training and Evaluation on Deep Reinforcement Learning for Automated Driving in Various Simulated Driving Maneuvers](https://mlsanity.com/paper/2306.11466/) (2023)
- [Evaluating Scenario-based Decision-making for Interactive Autonomous Driving Using Rational Criteria: A Survey](https://mlsanity.com/paper/2501.01886/) (2025)
- [Decision-Making under On-Ramp merge Scenarios by Distributional Soft Actor-Critic Algorithm](https://mlsanity.com/paper/2103.04535/) (2021)
- [Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving](https://mlsanity.com/paper/2111.05479/) (2021)
- [Delay-Aware Reinforcement Learning for Highway On-Ramp Merging under Stochastic Communication Latency](https://mlsanity.com/paper/2403.11852/) (2024)
- [Reinforced Imitative Trajectory Planning for Urban Automated Driving](https://mlsanity.com/paper/2410.15607/) (2024)
- [Dynamically Local-Enhancement Planner for Large-Scale Autonomous Driving](https://mlsanity.com/paper/2502.21134/) (2025)
- [Human-aligned Safe Reinforcement Learning for Highway On-Ramp Merging in Dense Traffic](https://mlsanity.com/paper/2503.02624/) (2025)
- [Adversarial Agent Behavior Learning in Autonomous Driving Using Deep Reinforcement Learning](https://mlsanity.com/paper/2508.15207/) (2025)
- [Beyond Safety Filtering: Control Barrier Function-Informed Reinforcement Learning for Connected and Automated Vehicles](https://mlsanity.com/paper/2605.16894/) (2026)

## Abstract (arXiv)

Multiple automakers have in development or in production automated driving systems (ADS) that offer freeway-pilot functions. This type of ADS is typically limited to restricted-access freeways only, that is, the transition from manual to automated modes takes place only after the ramp merging process is completed manually. One major challenge to extend the automation to ramp merging is that the automated vehicle needs to incorporate and optimize long-term objectives (e.g. successful and smooth merge) when near-term actions must be safely executed. Moreover, the merging process involves interactions with other vehicles whose behaviors are sometimes hard to predict but may influence the merging vehicle optimal actions. To tackle such a complicated control problem, we propose to apply Deep Reinforcement Learning (DRL) techniques for finding an optimal driving policy by maximizing the long-term reward in an interactive environment. Specifically, we apply a Long Short-Term Memory (LSTM) architecture to model the interactive environment, from which an internal state containing historical driving information is conveyed to a Deep Q-Network (DQN). The DQN is used to approximate the Q-function, which takes the internal state as input and generates Q-values as output for action selection. With this DRL architecture, the historical impact of interactive environment on the long-term reward can be captured and taken into account for deciding the optimal control policy. The proposed architecture has the potential to be extended and applied to other autonomous driving scenarios such as driving through a complex intersection or changing lanes under varying traffic flow conditions.