# Depth-SIMS: Semi-Parametric Image and Depth Synthesis

arXiv: https://arxiv.org/abs/2203.03405 · HTML: https://mlsanity.com/paper/2203.03405/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [A2D2: Audi Autonomous Driving Dataset](https://mlsanity.com/paper/2004.06320/) (2020)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [PENet: Towards Precise and Efficient Image Guided Depth Completion](https://mlsanity.com/paper/2103.00783/) (2021)
- [High Quality Monocular Depth Estimation via Transfer Learning](https://mlsanity.com/paper/1812.11941/) (2018)
- [You Only Need Adversarial Supervision for Semantic Image Synthesis](https://mlsanity.com/paper/2012.04781/) (2020)
- [Deep Image Harmonization](https://mlsanity.com/paper/1703.00069/) (2017)
- [GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving](https://mlsanity.com/paper/2101.06543/) (2021)
- [Monocular Depth Estimation Based On Deep Learning: An Overview](https://mlsanity.com/paper/2003.06620/) (2020)
- [SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation](https://mlsanity.com/paper/1810.01849/) (2018)
- [Semi-parametric Image Synthesis](https://mlsanity.com/paper/1804.10992/) (2018)
- [Context-Aware Synthesis and Placement of Object Instances](https://mlsanity.com/paper/1812.02350/) (2018)
- [DVI: Depth Guided Video Inpainting for Autonomous Driving](https://mlsanity.com/paper/2007.08854/) (2020)

## Abstract (arXiv)

In this paper we present a compositing image synthesis method that generates RGB canvases with well aligned segmentation maps and sparse depth maps, coupled with an in-painting network that transforms the RGB canvases into high quality RGB images and the sparse depth maps into pixel-wise dense depth maps. We benchmark our method in terms of structural alignment and image quality, showing an increase in mIoU over SOTA by 3.7 percentage points and a highly competitive FID. Furthermore, we analyse the quality of the generated data as training data for semantic segmentation and depth completion, and show that our approach is more suited for this purpose than other methods.