# Revisiting 3D Object Detection From an Egocentric Perspective

arXiv: https://arxiv.org/abs/2112.07787 · HTML: https://mlsanity.com/paper/2112.07787/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 69 references that are in the map.
- First posted on arXiv 14 Dec 2021.

## Builds on (most cited first, 30 of 51)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [STD: Sparse-to-Dense 3D Object Detector for Point Cloud](https://mlsanity.com/paper/1907.10471/) (2019)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis](https://mlsanity.com/paper/1612.00101/) (2016)
- [Multi-Task Multi-Sensor Fusion for 3D Object Detection](https://mlsanity.com/paper/2012.12397/) (2020)
- [Boundary IoU: Improving Object-Centric Image Segmentation Evaluation](https://mlsanity.com/paper/2103.16562/) (2021)
- [End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/1910.06528/) (2019)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [MVX-Net: Multimodal VoxelNet for 3D Object Detection](https://mlsanity.com/paper/1904.01649/) (2019)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [Pillar-based Object Detection for Autonomous Driving](https://mlsanity.com/paper/2007.10323/) (2020)
- [Fast Point R-CNN](https://mlsanity.com/paper/1908.02990/) (2019)

## Built on by (most cited first, 18 of 18)

- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [3D Object Detection from Images for Autonomous Driving: A Survey](https://mlsanity.com/paper/2202.02980/) (2022)
- [Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving](https://mlsanity.com/paper/2210.08061/) (2022)
- [Revisiting Domain-Adaptive 3D Object Detection by Reliable, Diverse and Class-balanced Pseudo-Labeling](https://mlsanity.com/paper/2307.07944/) (2023)
- [Exploring Active 3D Object Detection from a Generalization Perspective](https://mlsanity.com/paper/2301.09249/) (2023)
- [Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environments](https://mlsanity.com/paper/2403.13556/) (2024)
- [On Offline Evaluation of 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/2308.12779/) (2023)
- [Transcendental Idealism of Planner: Evaluating Perception from Planning Perspective for Autonomous Driving](https://mlsanity.com/paper/2306.07276/) (2023)
- [MOS: Model Synergy for Test-Time Adaptation on LiDAR-Based 3D Object Detection](https://mlsanity.com/paper/2406.14878/) (2024)
- [CG-SSD: Corner Guided Single Stage 3D Object Detection from LiDAR Point Cloud](https://mlsanity.com/paper/2202.11868/) (2022)
- [USC: Uncompromising Spatial Constraints for Safety-Oriented 3D Object Detectors in Autonomous Driving](https://mlsanity.com/paper/2209.10368/) (2022)
- [SURE-Val: Safe Urban Relevance Extension and Validation](https://mlsanity.com/paper/2308.02266/) (2023)
- [Instance-Level Safety-Aware Fidelity of Synthetic Data and Its Calibration](https://mlsanity.com/paper/2402.07031/) (2024)
- [EC-IoU: Orienting Safety for Object Detectors via Ego-Centric Intersection-over-Union](https://mlsanity.com/paper/2403.15474/) (2024)
- [You Only Label Once: 3D Box Adaptation from Point Cloud to Image via Semi-Supervised Learning](https://mlsanity.com/paper/2211.09302/) (2022)
- [STONE: A Submodular Optimization Framework for Active 3D Object Detection](https://mlsanity.com/paper/2410.03918/) (2024)
- [Open-CRB: Towards Open World Active Learning for 3D Object Detection](https://mlsanity.com/paper/2310.10391/) (2023)
- [Safety-Aligned 3D Object Detection: Single-Vehicle, Cooperative, and End-to-End Perspectives](https://mlsanity.com/paper/2604.03325/) (2026)

## Abstract (arXiv)

3D object detection is a key module for safety-critical robotics applications such as autonomous driving. For these applications, we care most about how the detections affect the ego-agent's behavior and safety (the egocentric perspective). Intuitively, we seek more accurate descriptions of object geometry when it's more likely to interfere with the ego-agent's motion trajectory. However, current detection metrics, based on box Intersection-over-Union (IoU), are object-centric and aren't designed to capture the spatio-temporal relationship between objects and the ego-agent. To address this issue, we propose a new egocentric measure to evaluate 3D object detection, namely Support Distance Error (SDE). Our analysis based on SDE reveals that the egocentric detection quality is bounded by the coarse geometry of the bounding boxes. Given the insight that SDE would benefit from more accurate geometry descriptions, we propose to represent objects as amodal contours, specifically amodal star-shaped polygons, and devise a simple model, StarPoly, to predict such contours. Our experiments on the large-scale Waymo Open Dataset show that SDE better reflects the impact of detection quality on the ego-agent's safety compared to IoU; and the estimated contours from StarPoly consistently improve the egocentric detection quality over recent 3D object detectors.