# Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving

arXiv: https://arxiv.org/abs/2311.10747 · HTML: https://mlsanity.com/paper/2311.10747/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 31 Oct 2023.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/)

## Builds on (most cited first, 30 of 33)

- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Batch Policy Learning under Constraints](https://mlsanity.com/paper/1903.08738/) (2019)
- [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022)
- [Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving](https://mlsanity.com/paper/2305.06242/) (2023)
- [A General Framework for Uncertainty Estimation in Deep Learning](https://mlsanity.com/paper/1907.06890/) (2019)
- [A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective](https://mlsanity.com/paper/2202.02215/) (2022)
- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning](https://mlsanity.com/paper/2001.08726/) (2020)
- [EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning](https://mlsanity.com/paper/1807.08364/) (2018)
- [Model-free Deep Reinforcement Learning for Urban Autonomous Driving](https://mlsanity.com/paper/1904.09503/) (2019)
- [Offline Reinforcement Learning from Images with Latent Space Models](https://mlsanity.com/paper/2012.11547/) (2020)
- [Constraints Penalized Q-learning for Safe Offline Reinforcement Learning](https://mlsanity.com/paper/2107.09003/) (2021)
- [Constrained Variational Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2201.11927/) (2022)
- [Constrained Decision Transformer for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2302.07351/) (2023)
- [Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning](https://mlsanity.com/paper/2008.07971/) (2020)
- [Datasets and Benchmarks for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2306.09303/) (2023)
- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning](https://mlsanity.com/paper/2207.09081/) (2022)
- [Context-Aware Safe Reinforcement Learning for Non-Stationary Environments](https://mlsanity.com/paper/2101.00531/) (2021)
- [Offline Reinforcement Learning with Pseudometric Learning](https://mlsanity.com/paper/2103.01948/) (2021)
- [Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation](https://mlsanity.com/paper/2307.07907/) (2023)
- [Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Driving](https://mlsanity.com/paper/2208.12263/) (2022)
- [CausalAF: Causal Autoregressive Flow for Safety-Critical Driving Scenario Generation](https://mlsanity.com/paper/2110.13939/) (2021)

## Built on by (most cited first, 17 of 17)

- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization](https://mlsanity.com/paper/2402.14528/) (2024)
- [Gradient Shaping for Multi-Constraint Safe Reinforcement Learning](https://mlsanity.com/paper/2312.15127/) (2023)
- [OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2407.14653/) (2024)
- [Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Driving](https://mlsanity.com/paper/2409.00858/) (2024)
- [Temporal Logic-Based Multi-Vehicle Backdoor Attacks against Offline RL Agents in End-to-end Autonomous Driving](https://mlsanity.com/paper/2509.16950/) (2025)
- [Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approach](https://mlsanity.com/paper/2312.07795/) (2023)
- [Feasibility Consistent Representation Learning for Safe Reinforcement Learning](https://mlsanity.com/paper/2405.11718/) (2024)
- [A Safe and Efficient Self-evolving Algorithm for Decision-making and Control of Autonomous Driving Systems](https://mlsanity.com/paper/2408.12187/) (2024)
- [Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossing](https://mlsanity.com/paper/2504.08704/) (2025)
- [Boundary-to-Region Supervision for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2509.25727/) (2025)
- [Online Optimization for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2510.22027/) (2025)
- [A Generalizable Physics-guided Causal Model for Trajectory Prediction in Autonomous Driving](https://mlsanity.com/paper/2602.13936/) (2026)
- [Intrinsic Dynamics-Driven Generalizable Scene Representations for Vision-Oriented Decision-Making Applications](https://mlsanity.com/paper/2405.19736/) (2024)
- [Generating Causal Explanations of Vehicular Agent Behavioural Interactions with Learnt Reward Profiles](https://mlsanity.com/paper/2503.14557/) (2025)
- [Safe Reinforcement Learning of Autonomous Highway Driving: A Unified Framework for Safety and Efficiency](https://mlsanity.com/paper/2606.14609/) (2026)
- [A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platform](https://mlsanity.com/paper/2610.00926/) (2026)

## Abstract (arXiv)

In the domain of autonomous driving, the offline Reinforcement Learning~(RL) approaches exhibit notable efficacy in addressing sequential decision-making problems from offline datasets. However, maintaining safety in diverse safety-critical scenarios remains a significant challenge due to long-tailed and unforeseen scenarios absent from offline datasets. In this paper, we introduce the saFety-aware strUctured Scenario representatION (FUSION), a pioneering representation learning method in offline RL to facilitate the learning of a generalizable end-to-end driving policy by leveraging structured scenario information. FUSION capitalizes on the causal relationships between the decomposed reward, cost, state, and action space, constructing a framework for structured sequential reasoning in dynamic traffic environments. We conduct extensive evaluations in two typical real-world settings of the distribution shift in autonomous vehicles, demonstrating the good balance between safety cost and utility reward compared to the current state-of-the-art safe RL and IL baselines. Empirical evidence in various driving scenarios attests that FUSION significantly enhances the safety and generalizability of autonomous driving agents, even in the face of challenging and unseen environments. Furthermore, our ablation studies reveal noticeable improvements in the integration of causal representation into the offline safe RL algorithm. Our code implementation is available at: https://sites.google.com/view/safe-fusion/.