# Decoupling Scene Perception and Ego Status: A Multi-Context Fusion Approach for Enhanced Generalization in End-to-End Autonomous Driving

arXiv: https://arxiv.org/abs/2511.13079 · HTML: https://mlsanity.com/paper/2511.13079/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 17 Nov 2025.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024)
- [Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability](https://mlsanity.com/paper/2405.17398/) (2024)
- [Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving](https://mlsanity.com/paper/2406.03877/) (2024)
- [SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation](https://mlsanity.com/paper/2405.19620/) (2024)
- [Rethinking Rotated Object Detection with Gaussian Wasserstein Distance Loss](https://mlsanity.com/paper/2101.11952/) (2021)
- [DeepInteraction: 3D Object Detection via Modality Interaction](https://mlsanity.com/paper/2208.11112/) (2022)
- [PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Driving](https://mlsanity.com/paper/2404.14327/) (2024)
- [FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Driving](https://mlsanity.com/paper/2308.01006/) (2023)
- [LaneSegNet: Map Learning with Lane Segment Perception for Autonomous Driving](https://mlsanity.com/paper/2312.16108/) (2023)
- [Bridging Past and Future: End-to-End Autonomous Driving with Historical Prediction and Planning](https://mlsanity.com/paper/2503.14182/) (2025)
- [BEVFusion4D: Learning LiDAR-Camera Fusion Under Bird's-Eye-View via Cross-Modality Guidance and Temporal Aggregation](https://mlsanity.com/paper/2303.17099/) (2023)
- [PointSSC: A Cooperative Vehicle-Infrastructure Point Cloud Benchmark for Semantic Scene Completion](https://mlsanity.com/paper/2309.12708/) (2023)

## Built on by (most cited first, 4 of 4)

- [CausalVAD: De-confounding End-to-End Autonomous Driving via Causal Intervention](https://mlsanity.com/paper/2603.18561/) (2026)
- [DINO-VO: Learning Where to Focus for Enhanced State Estimation](https://mlsanity.com/paper/2604.04055/) (2026)
- [Does Visual Information Play a Decisive Role in Vision-Language-Action Model Driving Behavior?](https://mlsanity.com/paper/2605.31041/) (2026)
- [Auditing Latent-Space Monitors for Autonomous Driving](https://mlsanity.com/paper/2609.30557/) (2026)

## Abstract (arXiv)

Modular design of planning-oriented autonomous driving has markedly advanced end-to-end systems. However, existing architectures remain constrained by an over-reliance on ego status, hindering generalization and robust scene understanding. We identify the root cause as an inherent design within these architectures that allows ego status to be easily leveraged as a shortcut. Specifically, the premature fusion of ego status in the upstream BEV encoder allows an information flow from this strong prior to dominate the downstream planning module. To address this challenge, we propose AdaptiveAD, an architectural-level solution based on a multi-context fusion strategy. Its core is a dual-branch structure that explicitly decouples scene perception and ego status. One branch performs scene-driven reasoning based on multi-task learning, but with ego status deliberately omitted from the BEV encoder, while the other conducts ego-driven reasoning based solely on the planning task. A scene-aware fusion module then adaptively integrates the complementary decisions from the two branches to form the final planning trajectory. To ensure this decoupling does not compromise multi-task learning, we introduce a path attention mechanism for ego-BEV interaction and add two targeted auxiliary tasks: BEV unidirectional distillation and autoregressive online mapping. Extensive evaluations on the nuScenes dataset demonstrate that AdaptiveAD achieves state-of-the-art open-loop planning performance. Crucially, it significantly mitigates the over-reliance on ego status and exhibits impressive generalization capabilities across diverse scenarios.