# See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selection arXiv: https://arxiv.org/abs/2601.10707 · HTML: https://mlsanity.com/paper/2601.10707/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. - Builds on 48 of its 56 references that are in the map. - First posted on arXiv 15 Jan 2026. ## Placed in - End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/) ## Builds on (most cited first, 30 of 48) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017) - [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021) - [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022) - [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022) - [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022) - [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016) - [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021) - [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022) - [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021) - [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022) - [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017) - [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021) - [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021) - [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023) - [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023) - [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022) - [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018) - [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020) - [Pre-Trained Language Models for Interactive Decision-Making](https://mlsanity.com/paper/2202.01771/) (2022) - [PLA: Language-Driven Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2211.16312/) (2022) - [End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances](https://mlsanity.com/paper/1911.10868/) (2019) ## Built on by (most cited first, 4 of 4) - [Robustness Is a Function, Not a Number: A Factorized Comprehensive Study of OOD Robustness in Vision-Based Driving](https://mlsanity.com/paper/2602.09018/) (2026) - [The Constant Eye: Benchmarking and Bridging Appearance Robustness in Autonomous Driving](https://mlsanity.com/paper/2602.12563/) (2026) - [Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fields](https://mlsanity.com/paper/2609.03117/) (2026) - [ConDenseDrive: Knowledge-Dense, Token-Sparse Visual Encoder for End-to-End Autonomous Driving](https://mlsanity.com/paper/2610.16908/) (2026) ## Abstract (arXiv) Recent advances in end-to-end autonomous driving show that policies trained on patch-aligned features extracted from foundation models generalize better to Out-of-Distribution (OOD). We hypothesize that due to the self-attention mechanism, each patch feature implicitly embeds/contains information from all other patches, represented in a different way and intensity, making these descriptors highly redundant. We quantify redundancy in such (BLIP2) features via PCA and cross-patch similarity: 90% of variance is captured by 17/64 principal components, and strong inter-token correlations are pervasive. Training on such overlapping information leads the policy to overfit spurious correlations, hurting OOD robustness. We present Stochastic-Patch-Selection (SPS), a simple yet effective approach for learning policies that are more robust, generalizable, and efficient. For every frame, SPS randomly masks a fraction of patch descriptors, not feeding them to the policy model, while preserving the spatial layout of the remaining patches. Thus, the policy is provided with different stochastic but complete views of the (same) scene: every random subset of patches acts like a different, yet still sensible, coherent projection of the world. The policy thus bases its decisions on features that are invariant to which specific tokens survive. Extensive experiments confirm that across all OOD scenarios, our method outperforms the state of the art (SOTA), achieving a 6.2% average improvement and up to 20.4% in closed-loop simulations, while being 2.4× faster. We conduct ablations over masking rates and patch-feature reorganization, training and evaluating 9 systems, with 8 of them surpassing prior SOTA. Finally, we show that the same learned policy transfers to a physical, real-world car without any tuning.