# SHIFT3D: Synthesizing Hard Inputs For Tricking 3D Detectors

arXiv: https://arxiv.org/abs/2309.05810 · HTML: https://mlsanity.com/paper/2309.05810/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 33 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 30 of 30)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Adversarial Examples for Semantic Segmentation and Object Detection](https://mlsanity.com/paper/1703.08603/) (2017)
- [Generating Adversarial Examples with Adversarial Networks](https://mlsanity.com/paper/1801.02610/) (2018)
- [Embracing Single Stride 3D Object Detector with Sparse Transformer](https://mlsanity.com/paper/2112.06375/) (2021)
- [Generating 3D Adversarial Point Clouds](https://mlsanity.com/paper/1809.07016/) (2018)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [Physically Realizable Adversarial Examples for LiDAR Object Detection](https://mlsanity.com/paper/2004.00543/) (2020)
- [Constructing Unrestricted Adversarial Examples with Generative Models](https://mlsanity.com/paper/1805.07894/) (2018)
- [PointCloud Saliency Maps](https://mlsanity.com/paper/1812.01687/) (2018)
- [Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks](https://mlsanity.com/paper/2106.09249/) (2021)
- [Extending Adversarial Attacks and Defenses to Deep 3D Point Cloud Classifiers](https://mlsanity.com/paper/1901.03006/) (2019)
- [Adversarial Attacks Beyond the Image Space](https://mlsanity.com/paper/1711.07183/) (2017)
- [Adversarial Attack and Defense on Point Sets](https://mlsanity.com/paper/1902.10899/) (2019)
- [Adversarial Objects Against LiDAR-Based Autonomous Driving Systems](https://mlsanity.com/paper/1907.05418/) (2019)
- [Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifiers](https://mlsanity.com/paper/1904.08489/) (2019)
- [Robustness of 3D Deep Learning in an Adversarial Setting](https://mlsanity.com/paper/1904.00923/) (2019)
- [LG-GAN: Label Guided Adversarial Network for Flexible Targeted Attack of Point Cloud-based Deep Networks](https://mlsanity.com/paper/2011.00566/) (2020)
- [ReDAL: Region-based and Diversity-aware Active Learning for Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2107.11769/) (2021)
- [Minimal Adversarial Examples for Deep Learning on 3D Point Clouds](https://mlsanity.com/paper/2008.12066/) (2020)
- [Active Learning at the ImageNet Scale](https://mlsanity.com/paper/2111.12880/) (2021)
- [SAMP: Shape and Motion Priors for 4D Vehicle Reconstruction](https://mlsanity.com/paper/2005.00922/) (2020)
- [Improving the Intra-class Long-tail in 3D Detection via Rare Example Mining](https://mlsanity.com/paper/2210.08375/) (2022)
- [GradTail: Learning Long-Tailed Data Using Gradient-based Sample Weighting](https://mlsanity.com/paper/2201.05938/) (2022)

## Built on by (most cited first, 1 of 1)

- [Just Add 100 More: Augmenting NeRF-based Pseudo-LiDAR Point Cloud for Resolving Class-imbalance Problem](https://mlsanity.com/paper/2403.11573/) (2024)

## Abstract (arXiv)

We present SHIFT3D, a differentiable pipeline for generating 3D shapes that are structurally plausible yet challenging to 3D object detectors. In safety-critical applications like autonomous driving, discovering such novel challenging objects can offer insight into unknown vulnerabilities of 3D detectors. By representing objects with a signed distanced function (SDF), we show that gradient error signals allow us to smoothly deform the shape or pose of a 3D object in order to confuse a downstream 3D detector. Importantly, the objects generated by SHIFT3D physically differ from the baseline object yet retain a semantically recognizable shape. Our approach provides interpretable failure modes for modern 3D object detectors, and can aid in preemptive discovery of potential safety risks within 3D perception systems before these risks become critical failures.