# A Variational U-Net for Conditional Appearance and Shape Generation

arXiv: https://arxiv.org/abs/1804.04694 · HTML: https://mlsanity.com/paper/1804.04694/

## Key facts

- Cited by 124 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2018.

## Built on by (most cited first, 30 of 124)

- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [StyleGAN-Human: A Data-Centric Odyssey of Human Generation](https://mlsanity.com/paper/2204.11823/) (2022)
- [A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction](https://mlsanity.com/paper/2108.06852/) (2021)
- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [Mutual Consistency Learning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2109.09960/) (2021)
- [Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control](https://mlsanity.com/paper/2106.02019/) (2021)
- [Person Image Synthesis via Denoising Diffusion Model](https://mlsanity.com/paper/2211.12500/) (2022)
- [Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models](https://mlsanity.com/paper/2310.06313/) (2023)
- [Exploring Dual-task Correlation for Pose Guided Person Image Generation](https://mlsanity.com/paper/2203.02910/) (2022)
- [Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations](https://mlsanity.com/paper/2204.08839/) (2022)
- [StylePeople: A Generative Model of Fullbody Human Avatars](https://mlsanity.com/paper/2104.08363/) (2021)
- [Neural Re-Rendering of Humans from a Single Image](https://mlsanity.com/paper/2101.04104/) (2021)
- [Neural Texture Extraction and Distribution for Controllable Person Image Synthesis](https://mlsanity.com/paper/2204.06160/) (2022)
- [Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editing](https://mlsanity.com/paper/2104.07021/) (2021)
- [Uncertainty Inspired Underwater Image Enhancement](https://mlsanity.com/paper/2207.09689/) (2022)
- [Style and Pose Control for Image Synthesis of Humans from a Single Monocular View](https://mlsanity.com/paper/2102.11263/) (2021)
- [Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects](https://mlsanity.com/paper/2112.11347/) (2021)
- [Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis](https://mlsanity.com/paper/2402.18078/) (2024)
- [HumanGAN: A Generative Model of Humans Images](https://mlsanity.com/paper/2103.06902/) (2021)
- [Learning Disentangled Representations in the Imaging Domain](https://mlsanity.com/paper/2108.12043/) (2021)
- [Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN](https://mlsanity.com/paper/2109.06166/) (2021)
- [Cross Attention Based Style Distribution for Controllable Person Image Synthesis](https://mlsanity.com/paper/2208.00712/) (2022)
- [Learning Semantic Person Image Generation by Region-Adaptive Normalization](https://mlsanity.com/paper/2104.06650/) (2021)
- [Generating Representative Samples for Few-Shot Classification](https://mlsanity.com/paper/2205.02918/) (2022)
- [Understanding Object Dynamics for Interactive Image-to-Video Synthesis](https://mlsanity.com/paper/2106.11303/) (2021)
- [Disentanglement of Correlated Factors via Hausdorff Factorized Support](https://mlsanity.com/paper/2210.07347/) (2022)
- [Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation](https://mlsanity.com/paper/2109.10595/) (2021)
- [ZFlow: Gated Appearance Flow-based Virtual Try-on with 3D Priors](https://mlsanity.com/paper/2109.07001/) (2021)
- [SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2111.13495/) (2021)
- [HumanGen: Generating Human Radiance Fields with Explicit Priors](https://mlsanity.com/paper/2212.05321/) (2022)

## Abstract (arXiv)

Deep generative models have demonstrated great performance in image synthesis. However, results deteriorate in case of spatial deformations, since they generate images of objects directly, rather than modeling the intricate interplay of their inherent shape and appearance. We present a conditional U-Net for shape-guided image generation, conditioned on the output of a variational autoencoder for appearance. The approach is trained end-to-end on images, without requiring samples of the same object with varying pose or appearance. Experiments show that the model enables conditional image generation and transfer. Therefore, either shape or appearance can be retained from a query image, while freely altering the other. Moreover, appearance can be sampled due to its stochastic latent representation, while preserving shape. In quantitative and qualitative experiments on COCO, DeepFashion, shoes, Market-1501 and handbags, the approach demonstrates significant improvements over the state-of-the-art.