Data to 7 Oct 2026

Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model

arXiv 2210.04017 ↗First posted 8 Oct 2022cs.LGcited by 44

On the map: World models → World models for autonomous driving

also in 1 more

End-to-end autonomous driving → Generative simulation for driving

Abstract (arXiv)

End-to-end autonomous driving provides a feasible way to automatically maximize overall driving system performance by directly mapping the raw pixels from a front-facing camera to control signals. Recent advanced methods construct a latent world model to map the high dimensional observations into compact latent space. However, the latent states embedded by the world model proposed in previous works may contain a large amount of task-irrelevant information, resulting in low sampling efficiency and poor robustness to input perturbations. Meanwhile, the training data distribution is usually unbalanced, and the learned policy is challenging to cope with the corner cases during the driving process. To solve the above challenges, we present a SEMantic Masked recurrent world model (SEM2), which introduces a semantic filter to extract key driving-relevant features and make decisions via the filtered features, and is trained with a multi-source data sampler, which aggregates common data and multiple corner case data in a single batch, to balance the data distribution. Extensive experiments on CARLA show our method outperforms the state-of-the-art approaches in terms of sample efficiency and robustness to input permutations.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 31 of its 55 references that are in the map. First posted on arXiv 8 Oct 2022.

Builds on

31 of 55 references.

Closest ancestors · 2

Same direction, newest first

2019Model-free Deep Reinforcement Learning for Urban Autonomous Driving20 Apr
2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 29

2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
2016End to End Learning for Self-Driving Carscited by 1,000
2020Mastering Atari with Discrete World Modelscited by 990
2015End-to-End Training of Deep Visuomotor Policiescited by 847
2018Recurrent World Models Facilitate Policy Evolutioncited by 761
2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
2017End-to-end Driving via Conditional Imitation Learningcited by 479

17 more

2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
2019Learning Lightweight Lane Detection CNNs by Self Attention Distillationcited by 162
2018Towards End-to-End Lane Detection: an Instance Segmentation Approachcited by 129
2019Multimodal End-to-End Autonomous Drivingcited by 110
2017Interpretable Learning for Self-Driving Cars by Visualizing Causal Attentioncited by 106
2020Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learningcited by 99
2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
2018Deep Reinforcement Learning for Autonomous Drivingcited by 36
2021Model-based Constrained Reinforcement Learning using Generalized Control Barrier Functioncited by 30
2021Integrated Decision and Control: Towards Interpretable and Computationally Efficient Driving Intelligencecited by 25
2020PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inferencecited by 22
2020Bridging Imagination and Reality for Model-Based Deep Reinforcement Learningcited by 15
2020Integrating Deep Reinforcement Learning with Model-based Path Planners for Automated Drivingcited by 7
2019Relaxed Actor-Critic with Convergence Guarantees for Continuous-Time Optimal Control of Nonlinear Systemscited by 4
2018Optimal Stochastic Vehicle Path Planning Using Covariance Steeringcited by 3
2021Steadily Learn to Drive with Virtual Memorycited by 3
2021Uncertainty-Aware Model-Based Reinforcement Learning with Application to Autonomous Drivingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 15; 2025: 17; 2026: 8.

44 papers.

Sort:

By area:

2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Drivingcited by 89
2024World Models for Autonomous Driving: An Initial Surveycited by 79
2025A Comprehensive Survey on World Models for Embodied AIcited by 65
2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47

32 more

Similar, not cited

Close in wording, no citation either way