# Transcendental Idealism of Planner: Evaluating Perception from Planning Perspective for Autonomous Driving arXiv: https://arxiv.org/abs/2306.07276 ยท HTML: https://mlsanity.com/paper/2306.07276/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. - Builds on 31 of its 53 references that are in the map. - First posted on arXiv 12 Jun 2023. ## Placed in - End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/) ## Builds on (most cited first, 30 of 31) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017) - [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018) - [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020) - [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016) - [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019) - [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019) - [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018) - [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021) - [Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection](https://mlsanity.com/paper/1908.09492/) (2019) - [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019) - [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019) - [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018) - [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016) - [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021) - [Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations](https://mlsanity.com/paper/2008.05930/) (2020) - [A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving](https://mlsanity.com/paper/2011.10671/) (2020) - [Baidu Apollo EM Motion Planner](https://mlsanity.com/paper/1807.08048/) (2018) - [Imitating Driver Behavior with Generative Adversarial Networks](https://mlsanity.com/paper/1701.06699/) (2017) - [An investigation of model-free planning](https://mlsanity.com/paper/1901.03559/) (2019) - [Exploring Simple 3D Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2108.10312/) (2021) - [PillarNeXt: Rethinking Network Designs for 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/2305.04925/) (2023) - [Learning to Evaluate Perception Models Using Planner-Centric Metrics](https://mlsanity.com/paper/2004.08745/) (2020) - [ProphNet: Efficient Agent-Centric Motion Forecasting with Anchor-Informed Proposals](https://mlsanity.com/paper/2303.12071/) (2023) - [Self-Supervised Pillar Motion Learning for Autonomous Driving](https://mlsanity.com/paper/2104.08683/) (2021) - [Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving](https://mlsanity.com/paper/2210.09539/) (2022) - [Search-based optimal motion planning for automated driving](https://mlsanity.com/paper/1803.04868/) (2018) - [Injecting Planning-Awareness into Prediction and Detection Evaluation](https://mlsanity.com/paper/2110.03270/) (2021) ## Built on by (most cited first, 9 of 9) - [DistillBEV: Boosting Multi-Camera 3D Object Detection with Cross-Modal Knowledge Distillation](https://mlsanity.com/paper/2309.15109/) (2023) - [GEDepth: Ground Embedding for Monocular Depth Estimation](https://mlsanity.com/paper/2309.09975/) (2023) - [On Offline Evaluation of 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/2308.12779/) (2023) - [Cross-Modal Self-Supervised Learning with Effective Contrastive Units for LiDAR Point Clouds](https://mlsanity.com/paper/2409.06827/) (2024) - [TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations](https://mlsanity.com/paper/2606.17386/) (2026) - [Checklist to Define the Identification of TP, FP, and FN Object Detections in Automated Driving](https://mlsanity.com/paper/2308.07106/) (2023) - [Expanding-and-Shrinking Binary Neural Networks](https://mlsanity.com/paper/2503.23709/) (2025) - [Wireless Communication as an Information Sensor for Multi-agent Cooperative Perception: A Survey](https://mlsanity.com/paper/2505.00747/) (2025) - [The Decision Value of Perception Compute](https://mlsanity.com/paper/2609.35910/) (2026) ## Abstract (arXiv) Evaluating the performance of perception modules in autonomous driving is one of the most critical tasks in developing the complex intelligent system. While module-level unit test metrics adopted from traditional computer vision tasks are feasible to some extent, it remains far less explored to measure the impact of perceptual noise on the driving quality of autonomous vehicles in a consistent and holistic manner. In this work, we propose a principled framework that provides a coherent and systematic understanding of the impact an error in the perception module imposes on an autonomous agent's planning that actually controls the vehicle. Specifically, the planning process is formulated as expected utility maximisation, where all input signals from upstream modules jointly provide a world state description, and the planner strives for the optimal action by maximising the expected utility determined by both world states and actions. We show that, under practical conditions, the objective function can be represented as an inner product between the world state description and the utility function in a Hilbert space. This geometric interpretation enables a novel way to analyse the impact of noise in world state estimation on planning and leads to a universal metric for evaluating perception. The whole framework resembles the idea of transcendental idealism in the classical philosophical literature, which gives the name to our approach.