# 3D Copy-Paste: Physically Plausible Object Insertion for Monocular 3D Detection

arXiv: https://arxiv.org/abs/2312.05277 · HTML: https://mlsanity.com/paper/2312.05277/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 42 references that are in the map.
- First posted on arXiv 8 Dec 2023.

## Builds on (most cited first, 30 of 34)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [M3D-RPN: Monocular 3D Region Proposal Network for Object Detection](https://mlsanity.com/paper/1907.06038/) (2019)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [Disentangling Monocular 3D Object Detection](https://mlsanity.com/paper/1905.12365/) (2019)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation](https://mlsanity.com/paper/2002.10111/) (2020)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [Probabilistic and Geometric Depth: Detecting Objects in Perspective](https://mlsanity.com/paper/2107.14160/) (2021)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection](https://mlsanity.com/paper/2106.01178/) (2021)
- [RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving](https://mlsanity.com/paper/2001.03343/) (2020)
- [Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes](https://mlsanity.com/paper/1708.01566/) (2017)
- [3D Common Corruptions and Data Augmentation](https://mlsanity.com/paper/2203.01441/) (2022)
- [CubeSLAM: Monocular 3D Object SLAM](https://mlsanity.com/paper/1806.00557/) (2018)
- [Holistic 3D Scene Understanding from a Single Image with Implicit Representation](https://mlsanity.com/paper/2103.06422/) (2021)
- [Deep Parametric Indoor Lighting Estimation](https://mlsanity.com/paper/1910.08812/) (2019)
- [Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing](https://mlsanity.com/paper/2211.03017/) (2022)
- [GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving](https://mlsanity.com/paper/2101.06543/) (2021)
- [Vote3Deep: Fast Object Detection in 3D Point Clouds Using Efficient Convolutional Neural Networks](https://mlsanity.com/paper/1609.06666/) (2016)
- [Improved Adversarial Systems for 3D Object Generation and Reconstruction](https://mlsanity.com/paper/1707.09557/) (2017)
- [Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion](https://mlsanity.com/paper/2208.09480/) (2022)
- [Neural-Sim: Learning to Generate Training Data with NeRF](https://mlsanity.com/paper/2207.11368/) (2022)

## Built on by (most cited first, 15 of 15)

- [HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generation](https://mlsanity.com/paper/2503.16848/) (2025)
- [Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detection](https://mlsanity.com/paper/2406.00830/) (2024)
- [ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion](https://mlsanity.com/paper/2410.08168/) (2024)
- [BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation](https://mlsanity.com/paper/2405.09546/) (2024)
- [Unsupervised Discovery of Object-Centric Neural Fields](https://mlsanity.com/paper/2402.07376/) (2024)
- [Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environments](https://mlsanity.com/paper/2407.21452/) (2024)
- [AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring](https://mlsanity.com/paper/2501.09428/) (2025)
- [How far can we go with ImageNet for Text-to-Image generation?](https://mlsanity.com/paper/2502.21318/) (2025)
- [DreamView: Injecting View-specific Text Guidance into Text-to-3D Generation](https://mlsanity.com/paper/2404.06119/) (2024)
- [LabelAny3D: Label Any Object 3D in the Wild](https://mlsanity.com/paper/2601.01676/) (2026)
- [MonoPlace3D: Learning 3D-Aware Object Placement for 3D Monocular Detection](https://mlsanity.com/paper/2504.06801/) (2025)
- [Direct 3D-Aware Object Insertion via Decomposed Visual Proxies](https://mlsanity.com/paper/2606.06601/) (2026)
- [Object-Scene-Camera Decomposition and Recomposition for Data-Efficient Monocular 3D Object Detection](https://mlsanity.com/paper/2602.20627/) (2026)
- [3DRot: Rediscovering the Missing Primitive for RGB-Based 3D Augmentation](https://mlsanity.com/paper/2508.01423/) (2025)
- [Object-Centric Data Synthesis for Category-level Object Detection](https://mlsanity.com/paper/2511.23450/) (2025)

## Abstract (arXiv)

A major challenge in monocular 3D object detection is the limited diversity and quantity of objects in real datasets. While augmenting real scenes with virtual objects holds promise to improve both the diversity and quantity of the objects, it remains elusive due to the lack of an effective 3D object insertion method in complex real captured scenes. In this work, we study augmenting complex real indoor scenes with virtual objects for monocular 3D object detection. The main challenge is to automatically identify plausible physical properties for virtual assets (e.g., locations, appearances, sizes, etc.) in cluttered real scenes. To address this challenge, we propose a physically plausible indoor 3D object insertion approach to automatically copy virtual objects and paste them into real scenes. The resulting objects in scenes have 3D bounding boxes with plausible physical locations and appearances. In particular, our method first identifies physically feasible locations and poses for the inserted objects to prevent collisions with the existing room layout. Subsequently, it estimates spatially-varying illumination for the insertion location, enabling the immersive blending of the virtual objects into the original scene with plausible appearances and cast shadows. We show that our augmentation method significantly improves existing monocular 3D object models and achieves state-of-the-art performance. For the first time, we demonstrate that a physically plausible 3D object insertion, serving as a generative data augmentation technique, can lead to significant improvements for discriminative downstream tasks such as monocular 3D object detection. Project website: https://gyhandy.github.io/3D-Copy-Paste/