# Generalized Multi-Objective Reinforcement Learning with Envelope Updates in URLLC-enabled Vehicular Networks

arXiv: https://arxiv.org/abs/2405.11331 · HTML: https://mlsanity.com/paper/2405.11331/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Cooperative and V2X driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 19 of its 61 references that are in the map.
- First posted on arXiv 18 May 2024.

## Placed in

- End-to-end autonomous driving / [Cooperative and V2X driving](https://mlsanity.com/driving/cooperative-and-v2x-driving/)
- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [Urban Driving with Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1811.08586/) (2018)
- [Automated Lane Change Decision Making using Deep Reinforcement Learning in Dynamic and Uncertain Highway Environment](https://mlsanity.com/paper/1909.11538/) (2019)
- [Evolutionary Multi-Objective Reinforcement Learning Based Trajectory Control and Task Offloading in UAV-Assisted Mobile Edge Computing](https://mlsanity.com/paper/2202.12028/) (2022)
- [A Generative Car-following Model Conditioned On Driving Styles](https://mlsanity.com/paper/2112.05399/) (2021)
- [Reinforcement Learning for Joint V2I Network Selection and Autonomous Driving Policies](https://mlsanity.com/paper/2208.02249/) (2022)
- [Multi-UAV Speed Control with Collision Avoidance and Handover-aware Cell Association: DRL with Action Branching](https://mlsanity.com/paper/2307.13158/) (2023)
- [Detection and Imputation based Two-Stage Denoising Diffusion Power System Measurement Recovery under Cyber-Physical Uncertainties](https://mlsanity.com/paper/2312.04346/) (2023)
- [Mobility-Aware Performance in Hybrid RF and Terahertz Wireless Networks](https://mlsanity.com/paper/2112.10249/) (2021)
- [Reinforcement Learning Based Vehicle-cell Association Algorithm for Highly Mobile Millimeter Wave Communication](https://mlsanity.com/paper/2001.07915/) (2020)
- [Interference and Coverage Analysis in Coexisting RF and Dense TeraHertz Wireless Networks](https://mlsanity.com/paper/2006.08778/) (2020)
- [Optimization of Speed and Network Deployment for Reliable V2I Communication in the Presence of Handoffs and Interference](https://mlsanity.com/paper/2307.06852/) (2023)
- [Non-orthogonal Age-Optimal Information Dissemination in Vehicular Networks: A Meta Multi-Objective Reinforcement Learning Approach](https://mlsanity.com/paper/2402.12260/) (2024)
- [Comprehensive Training and Evaluation on Deep Reinforcement Learning for Automated Driving in Various Simulated Driving Maneuvers](https://mlsanity.com/paper/2306.11466/) (2023)
- [A Tractable Handoff-aware Rate Outage Approximation with Applications to THz-enabled Vehicular Network Optimization](https://mlsanity.com/paper/2308.03676/) (2023)

## Built on by (most cited first, 5 of 5)

- [Hierarchical and Collaborative LLM-Based Control for Multi-UAV Motion and Communication in Integrated Terrestrial and Non-Terrestrial Networks](https://mlsanity.com/paper/2506.06532/) (2025)
- [CVaR-Based Variational Quantum Optimization for User Association in Handoff-Aware Vehicular Networks](https://mlsanity.com/paper/2501.08418/) (2025)
- [Hybrid LLM-DDQN based Joint Optimization of V2I Communication and Autonomous Driving](https://mlsanity.com/paper/2410.08854/) (2024)
- [Hierarchical LLM-Driven Control for HAPS-Assisted UAV Networks: Joint Optimization of Flight and Connectivity](https://mlsanity.com/paper/2605.11509/) (2026)
- [Intelligent Multi-UAV Navigation in ITNTNs: A Hierarchical LLM Approach](https://mlsanity.com/paper/2607.18604/) (2026)

## Abstract (arXiv)

We develop a novel multi-objective reinforcement learning (MORL) framework to jointly optimize wireless network selection and autonomous driving policies in a multi-band vehicular network operating on conventional sub-6GHz spectrum and Terahertz frequencies. The proposed framework is designed to 1. maximize the traffic flow and minimize collisions by controlling the vehicle's motion dynamics (i.e., speed and acceleration), and 2. enhance the ultra-reliable low-latency communication (URLLC) while minimizing handoffs (HOs). We cast this problem as a multi-objective Markov Decision Process (MOMDP) and develop solutions for both predefined and unknown preferences of the conflicting objectives. Specifically, we develop a novel envelope MORL solution which develops policies that address multiple objectives with unknown preferences to the agent. While this approach reduces reliance on scalar rewards, policy effectiveness varying with different preferences is a challenge. To address this, we apply a generalized version of the Bellman equation and optimize the convex envelope of multi-objective Q values to learn a unified parametric representation capable of generating optimal policies across all possible preference configurations. Following an initial learning phase, our agent can execute optimal policies under any specified preference or infer preferences from minimal data samples. Numerical results validate the efficacy of the envelope-based MORL solution and demonstrate interesting insights related to the inter-dependency of vehicle motion dynamics, HOs, and the communication data rate. The proposed policies enable autonomous vehicles (AVs) to adopt safe driving behaviors with improved connectivity.