# Anytime 3D Object Reconstruction using Multi-modal Variational Autoencoder

arXiv: https://arxiv.org/abs/2101.10391 · HTML: https://mlsanity.com/paper/2101.10391/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 25 Jan 2021.

## Builds on (most cited first, 20 of 20)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [Generative and Discriminative Voxel Modeling with Convolutional Neural Networks](https://mlsanity.com/paper/1608.04236/) (2016)
- [MarrNet: 3D Shape Reconstruction via 2.5D Sketches](https://mlsanity.com/paper/1711.03129/) (2017)
- [CubeSLAM: Monocular 3D Object SLAM](https://mlsanity.com/paper/1806.00557/) (2018)
- [Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views](https://mlsanity.com/paper/1505.05641/) (2015)
- [CoReNet: Coherent 3D scene reconstruction from a single RGB image](https://mlsanity.com/paper/2004.12989/) (2020)
- [Image2Mesh: A Learning Framework for Single Image 3D Reconstruction](https://mlsanity.com/paper/1711.10669/) (2017)
- [Improving Missing Data Imputation with Deep Generative Models](https://mlsanity.com/paper/1902.10666/) (2019)
- [Zero-shot Learning via Simultaneous Generating and Learning](https://mlsanity.com/paper/1910.09446/) (2019)
- [Instant recovery of shape from spectrum via latent space connections](https://mlsanity.com/paper/2003.06523/) (2020)
- [Constructing Category-Specific Models for Monocular Object-SLAM](https://mlsanity.com/paper/1802.09292/) (2018)
- [A Variational Observation Model of 3D Object for Probabilistic Semantic SLAM](https://mlsanity.com/paper/1809.05225/) (2018)
- [Missing Features Reconstruction Using a Wasserstein Generative Adversarial Imputation Network](https://mlsanity.com/paper/2006.11783/) (2020)
- [A Variational Feature Encoding Method of 3D Object for Probabilistic Semantic SLAM](https://mlsanity.com/paper/1808.10180/) (2018)

## Built on by (most cited first, 2 of 2)

- [GARNet: Global-Aware Multi-View 3D Reconstruction Network and the Cost-Performance Tradeoff](https://mlsanity.com/paper/2211.02299/) (2022)
- [NegoCollab: A Common Representation Negotiation Approach for Heterogeneous Collaborative Perception](https://mlsanity.com/paper/2510.27647/) (2025)

## Abstract (arXiv)

For effective human-robot teaming, it is important for the robots to be able to share their visual perception with the human operators. In a harsh remote collaboration setting, data compression techniques such as autoencoder can be utilized to obtain and transmit the data in terms of latent variables in a compact form. In addition, to ensure real-time runtime performance even under unstable environments, an anytime estimation approach is desired that can reconstruct the full contents from incomplete information. In this context, we propose a method for imputation of latent variables whose elements are partially lost. To achieve the anytime property with only a few dimensions of variables, exploiting prior information of the category-level is essential. A prior distribution used in variational autoencoders is simply assumed to be isotropic Gaussian regardless of the labels of each training datapoint. This type of flattened prior makes it difficult to perform imputation from the category-level distributions. We overcome this limitation by exploiting a category-specific multi-modal prior distribution in the latent space. The missing elements of the partially transferred data can be sampled, by finding a specific modal according to the remaining elements. Since the method is designed to use partial elements for anytime estimation, it can also be applied for data over-compression. Based on the experiments on the ModelNet and Pascal3D datasets, the proposed approach shows consistently superior performance over autoencoder and variational autoencoder up to 70% data loss.