mlsanityData to 14 Oct 2026

Dreaming Across Towns: Semantic Rollout and Town-Adversarial Regularization for Zero-Shot Held-Out-Town Fixed-Route Driving in CARLA

arXiv 2604.27994 ↗First posted 30 Apr 2026cs.ROcited by 0

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

Abstract (arXiv)

Driving agents trained in one simulated town often perform poorly in a new town because the road shapes, intersections, and lane layouts can be different. This paper studies how to improve this kind of transfer in the CARLA driving simulator without giving the agent any training data from the test towns. The agent is trained only in Town05 and Town06, then evaluated directly in Town03 and Town04. To focus on road-layout differences, all experiments use the same weather and traffic settings. We propose a training method that encourages the agent to learn features that are useful across towns rather than features tied to one training town. During training, the agent is asked to predict the high-level visual meaning of future camera views and is also discouraged from relying on cues that reveal which source town the data came from. These extra learning signals are used only during training; at test time, the driving policy uses the same observation and control interface as the baseline agent. In controlled comparisons with matched DreamerV3-style world-model driving agents, the proposed method achieves the highest mean held-out success: 36.6% on Town03 with a 95% confidence interval of [30.5, 42.7] and 85.6% on Town04 with a 95% confidence interval of [84.0, 87.2], computed across five training seeds. Seed-paired tests against the strongest primary baselines show positive success-rate differences in both held-out towns. Additional experiments show that predicting future visual meaning alone or removing town-specific cues alone is not enough to match the combined method. These results suggest that combining future-scene understanding with reduced reliance on source-town-specific features can improve cross-town driving performance in this CARLA setting.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 30 of its 37 references that are in the map. First posted on arXiv 30 Apr 2026.

Builds on

30 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)22 May
  2. 2022Model-Based Imitation Learning for Urban Driving14 Oct

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  3. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  4. 2019Invariant Risk Minimizationcited by 1,854
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  8. 2023Mastering Diverse Domains through World Modelscited by 1,253
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  10. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  11. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  12. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
16 more
  1. 2019Learning by Cheatingcited by 383
  2. 2016Domain Separation Networkscited by 320
  3. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 149
  4. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  5. 2021Learning to drive from a world on railscited by 114
  6. 2021GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Drivingcited by 60
  7. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Modelcited by 45
  8. 2020Label Efficient Visual Abstractions for Autonomous Drivingcited by 33
  9. 2023Learning World Models with Identifiable Factorizationcited by 23
  10. 2025Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptationcited by 14
  11. 2023CLIP-Motion: Learning Reward Functions for Robotic Actions Using Consecutive Observationscited by 9
  12. 2025A Comprehensive Review of Reinforcement Learning for Autonomous Driving in the CARLA Simulatorcited by 7
  13. 2024CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learningcited by 5
  14. 2023RLAD: Reinforcement Learning from Pixels for Autonomous Driving in Urban Environmentscited by 4
  15. 2025Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policiescited by 2
  16. 2025InDRiVE: Reward-Free World-Model Pretraining for Autonomous Driving via Latent Disagreementcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving: Self-Supervised versus Supervised Representations
  2. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients
  3. 2025RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies
  4. 2025Toward Efficient and Robust Behavior Models for Multi-Agent Driving Simulation
  5. 2026Video2Track: From Real-World Interaction Videos to Steerable Adversarial Closed-Track Testing for Automated Driving Systems
  6. 2024Guiding Attention in End-to-End Driving Models
  7. 2019Urban Driving with Conditional Imitation Learning
  8. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coach