# ID-Unet: Iterative Soft and Hard Deformation for View Synthesis

arXiv: https://arxiv.org/abs/2103.02264 · HTML: https://mlsanity.com/paper/2103.02264/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 3 Mar 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [Toward Multimodal Image-to-Image Translation](https://mlsanity.com/paper/1711.11586/) (2017)
- [Geometric GAN](https://mlsanity.com/paper/1705.02894/) (2017)
- [View Synthesis by Appearance Flow](https://mlsanity.com/paper/1605.03557/) (2016)
- [Deformable GANs for Pose-based Human Image Generation](https://mlsanity.com/paper/1801.00055/) (2017)
- [A Variational U-Net for Conditional Appearance and Shape Generation](https://mlsanity.com/paper/1804.04694/) (2018)
- [Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks](https://mlsanity.com/paper/1805.07925/) (2018)
- [Transformation-Grounded Image Generation Network for Novel 3D View Synthesis](https://mlsanity.com/paper/1703.02921/) (2017)
- [ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributes](https://mlsanity.com/paper/1803.10562/) (2018)
- [CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training](https://mlsanity.com/paper/1703.10155/) (2017)
- [Positional Normalization](https://mlsanity.com/paper/1907.04312/) (2019)
- [Learning to Generate Chairs, Tables and Cars with Convolutional Networks](https://mlsanity.com/paper/1411.5928/) (2014)
- [CR-GAN: Learning Complete Representations for Multi-view Generation](https://mlsanity.com/paper/1806.11191/) (2018)
- [Disentangling Latent Space for VAE by Label Relevant/Irrelevant Dimensions](https://mlsanity.com/paper/1812.09502/) (2018)
- [Novel Views of Objects from a Single Image](https://mlsanity.com/paper/1602.00328/) (2016)
- [Novel View Synthesis on Unpaired Data by Conditional Deformable Variational Auto-Encoder](https://mlsanity.com/paper/2007.10618/) (2020)

## Built on by (most cited first, 2 of 2)

- [DGFont++: Robust Deformable Generative Networks for Unsupervised Font Generation](https://mlsanity.com/paper/2212.14742/) (2022)
- [Weakly Supervised Scene Text Generation for Low-resource Languages](https://mlsanity.com/paper/2306.14269/) (2023)

## Abstract (arXiv)

View synthesis is usually done by an autoencoder, in which the encoder maps a source view image into a latent content code, and the decoder transforms it into a target view image according to the condition. However, the source contents are often not well kept in this setting, which leads to unnecessary changes during the view translation. Although adding skipped connections, like Unet, alleviates the problem, but it often causes the failure on the view conformity. This paper proposes a new architecture by performing the source-to-target deformation in an iterative way. Instead of simply incorporating the features from multiple layers of the encoder, we design soft and hard deformation modules, which warp the encoder features to the target view at different resolutions, and give results to the decoder to complement the details. Particularly, the current warping flow is not only used to align the feature of the same resolution, but also as an approximation to coarsely deform the high resolution feature. Then the residual flow is estimated and applied in the high resolution, so that the deformation is built up in the coarse-to-fine fashion. To better constrain the model, we synthesize a rough target view image based on the intermediate flows and their warped features. The extensive ablation studies and the final results on two different data sets show the effectiveness of the proposed model.