# Deep Object-Centric Policies for Autonomous Driving

arXiv: https://arxiv.org/abs/1811.05432 · HTML: https://mlsanity.com/paper/1811.05432/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2018.

## Built on by (most cited first, 30 of 41)

- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors](https://mlsanity.com/paper/2210.11339/) (2022)
- [DRAMA: Joint Risk Localization and Captioning in Driving](https://mlsanity.com/paper/2209.10767/) (2022)
- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [Concept Bottleneck Model with Additional Unsupervised Concepts](https://mlsanity.com/paper/2202.01459/) (2022)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [Learning Generalizable Manipulation Policies with Object-Centric 3D Representations](https://mlsanity.com/paper/2310.14386/) (2023)
- [HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Driving](https://mlsanity.com/paper/2309.05186/) (2023)
- [Vision-based Manipulation from Single Human Video with Open-World Object Graphs](https://mlsanity.com/paper/2405.20321/) (2024)
- [Deep Imitation Learning for Autonomous Driving in Generic Urban Scenarios with Enhanced Safety](https://mlsanity.com/paper/1903.00640/) (2019)
- [ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Models](https://mlsanity.com/paper/2506.16211/) (2025)
- [Contrastive Domain Adaptation](https://mlsanity.com/paper/2103.15566/) (2021)
- [Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations](https://mlsanity.com/paper/2408.04380/) (2024)
- [Grounding Visual Representations with Texts for Domain Generalization](https://mlsanity.com/paper/2207.10285/) (2022)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [Learning by Watching](https://mlsanity.com/paper/2106.05966/) (2021)
- [Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations](https://mlsanity.com/paper/2205.06333/) (2022)
- [Object-Centric Instruction Augmentation for Robotic Manipulation](https://mlsanity.com/paper/2401.02814/) (2024)
- [Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Models](https://mlsanity.com/paper/2404.13474/) (2024)
- [Language-Guided Object-Centric Diffusion Policy for Generalizable and Collision-Aware Robotic Manipulation](https://mlsanity.com/paper/2407.00451/) (2024)
- [Behavioral Research and Practical Models of Drivers' Attention](https://mlsanity.com/paper/2104.05677/) (2021)
- [DROID: Driver-centric Risk Object Identification](https://mlsanity.com/paper/2106.13201/) (2021)
- [Driving through the Concept Gridlock: Unraveling Explainability Bottlenecks in Automated Driving](https://mlsanity.com/paper/2310.16639/) (2023)
- [Ego-Vehicle Action Recognition based on Semi-Supervised Contrastive Learning](https://mlsanity.com/paper/2303.00977/) (2023)
- [Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Driving](https://mlsanity.com/paper/2403.12176/) (2024)
- [OCRA: Object-Centric Learning with 3D and Tactile Priors for Human-to-Robot Action Transfer](https://mlsanity.com/paper/2603.14401/) (2026)
- [Instance-Aware Predictive Navigation in Multi-Agent Environments](https://mlsanity.com/paper/2101.05893/) (2021)
- [Automatic Curricula via Expert Demonstrations](https://mlsanity.com/paper/2106.09159/) (2021)
- [Adversarial Mixture Density Networks: Learning to Drive Safely from Collision Data](https://mlsanity.com/paper/2107.04485/) (2021)

## Abstract (arXiv)

While learning visuomotor skills in an end-to-end manner is appealing, deep neural networks are often uninterpretable and fail in surprising ways. For robotics tasks, such as autonomous driving, models that explicitly represent objects may be more robust to new scenes and provide intuitive visualizations. We describe a taxonomy of "object-centric" models which leverage both object instances and end-to-end learning. In the Grand Theft Auto V simulator, we show that object-centric models outperform object-agnostic methods in scenes with other vehicles and pedestrians, even with an imperfect detector. We also demonstrate that our architectures perform well on real-world environments by evaluating on the Berkeley DeepDrive Video dataset, where an object-centric model outperforms object-agnostic models in the low-data regimes.