# DSDNet: Deep Structured self-Driving Network

arXiv: https://arxiv.org/abs/2008.06041 · HTML: https://mlsanity.com/paper/2008.06041/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2020.

## Built on by (most cited first, 30 of 62)

- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion](https://mlsanity.com/paper/2306.03083/) (2023)
- [Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving](https://mlsanity.com/paper/2305.06242/) (2023)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting](https://mlsanity.com/paper/2101.06653/) (2021)
- [Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey](https://mlsanity.com/paper/2307.04370/) (2023)
- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [ADAPT: Action-aware Driving Caption Transformer](https://mlsanity.com/paper/2302.00673/) (2023)
- [GoRela: Go Relative for Viewpoint-Invariant Motion Forecasting](https://mlsanity.com/paper/2211.02545/) (2022)
- [A Survey of World Models for Autonomous Driving](https://mlsanity.com/paper/2501.11260/) (2025)
- [STAMP: Scalable Task And Model-agnostic Collaborative Perception](https://mlsanity.com/paper/2501.18616/) (2025)
- [End-to-End Autonomous Driving through V2X Cooperation](https://mlsanity.com/paper/2404.00717/) (2024)
- [JFP: Joint Future Prediction with Interactive Multi-Agent Modeling for Autonomous Driving](https://mlsanity.com/paper/2212.08710/) (2022)
- [Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perception](https://mlsanity.com/paper/2308.16714/) (2023)
- [Divide-and-Conquer for Lane-Aware Diverse Trajectory Prediction](https://mlsanity.com/paper/2104.08277/) (2021)
- [DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles](https://mlsanity.com/paper/2212.06437/) (2022)
- [Rethinking Closed-loop Training for Autonomous Driving](https://mlsanity.com/paper/2306.15713/) (2023)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillation](https://mlsanity.com/paper/2104.12861/) (2021)
- [LangCoop: Collaborative Driving with Language](https://mlsanity.com/paper/2504.13406/) (2025)
- [Deep Multi-Task Learning for Joint Localization, Perception, and Prediction](https://mlsanity.com/paper/2101.06720/) (2021)
- [MapPrior: Bird's-Eye View Map Layout Estimation with Generative Models](https://mlsanity.com/paper/2308.12963/) (2023)
- [Deep Structured Reactive Planning](https://mlsanity.com/paper/2101.06832/) (2021)
- [End-to-End Interactive Prediction and Planning with Optical Flow Distillation for Autonomous Driving](https://mlsanity.com/paper/2104.08862/) (2021)
- [CoCMT: Communication-Efficient Cross-Modal Transformer for Collaborative Perception](https://mlsanity.com/paper/2503.13504/) (2025)
- [Decoder Fusion RNN: Context and Interaction Aware Decoders for Trajectory Prediction](https://mlsanity.com/paper/2108.05814/) (2021)
- [QuAD: Query-based Interpretable Neural Motion Planning for Autonomous Driving](https://mlsanity.com/paper/2404.01486/) (2024)

## Abstract (arXiv)

In this paper, we propose the Deep Structured self-Driving Network (DSDNet), which performs object detection, motion prediction, and motion planning with a single neural network. Towards this goal, we develop a deep structured energy based model which considers the interactions between actors and produces socially consistent multimodal future predictions. Furthermore, DSDNet explicitly exploits the predicted future distributions of actors to plan a safe maneuver by using a structured planning cost. Our sample-based formulation allows us to overcome the difficulty in probabilistic inference of continuous random variables. Experiments on a number of large-scale self driving datasets demonstrate that our model significantly outperforms the state-of-the-art.