# Topologically-Aware Deformation Fields for Single-View 3D Reconstruction

arXiv: https://arxiv.org/abs/2205.06267 · HTML: https://mlsanity.com/paper/2205.06267/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 80 references that are in the map.
- First posted on arXiv 12 May 2022.

## Builds on (most cited first, 30 of 53)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](https://mlsanity.com/paper/2104.00677/) (2021)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)

## Built on by (most cited first, 29 of 29)

- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors](https://mlsanity.com/paper/2306.17843/) (2023)
- [DreamGaussian4D: Generative 4D Gaussian Splatting](https://mlsanity.com/paper/2312.17142/) (2023)
- [SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields](https://mlsanity.com/paper/2211.09786/) (2022)
- [SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields](https://mlsanity.com/paper/2212.02501/) (2022)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion](https://mlsanity.com/paper/2211.11674/) (2022)
- [Real3D: Scaling Up Large Reconstruction Models with Real-World Images](https://mlsanity.com/paper/2406.08479/) (2024)
- [CSDN: Cross-modal Shape-transfer Dual-refinement Network for Point Cloud Completion](https://mlsanity.com/paper/2208.00751/) (2022)
- [Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videos](https://mlsanity.com/paper/2312.13604/) (2023)
- [Disco4D: Disentangled 4D Human Generation and Animation from a Single Image](https://mlsanity.com/paper/2409.17280/) (2024)
- [Tree-D Fusion: Simulation-Ready Tree Dataset from Single Images with Diffusion Priors](https://mlsanity.com/paper/2407.10330/) (2024)
- [Single-view 3D Mesh Reconstruction for Seen and Unseen Categories](https://mlsanity.com/paper/2208.02676/) (2022)
- [Multi-view 3D Object Reconstruction and Uncertainty Modelling with Neural Shape Prior](https://mlsanity.com/paper/2306.11739/) (2023)
- [textttNePhi: Neural Deformation Fields for Approximately Diffeomorphic Medical Image Registration](https://mlsanity.com/paper/2309.07322/) (2023)
- [NAISR: A 3D Neural Additive Model for Interpretable Shape Representation](https://mlsanity.com/paper/2303.09234/) (2023)
- [Boosting3D: High-Fidelity Image-to-3D by Boosting 2D Diffusion Prior to 3D Prior with Progressive Learning](https://mlsanity.com/paper/2311.13617/) (2023)
- [LIST: Learning Implicitly from Spatial Transformers for Single-View 3D Reconstruction](https://mlsanity.com/paper/2307.12194/) (2023)
- [BuilDiff: 3D Building Shape Generation using Single-Image Conditional Point Cloud Diffusion Models](https://mlsanity.com/paper/2309.00158/) (2023)
- [The More You See in 2D, the More You Perceive in 3D](https://mlsanity.com/paper/2404.03652/) (2024)
- [3D Surface Reconstruction in the Wild by Deforming Shape Priors from Synthetic Data](https://mlsanity.com/paper/2302.12883/) (2023)
- [CD2: Fine-grained 3D Mesh Reconstruction With Twice Chamfer Distance](https://mlsanity.com/paper/2206.00447/) (2022)
- [Parcel3D: Shape Reconstruction from Single RGB Images for Applications in Transportation Logistics](https://mlsanity.com/paper/2304.08994/) (2023)
- [Vista3D: Unravel the 3D Darkside of a Single Image](https://mlsanity.com/paper/2409.12193/) (2024)
- [3DMiner: Discovering Shapes from Large-Scale Unannotated Image Datasets](https://mlsanity.com/paper/2310.19188/) (2023)
- [Single-image coherent reconstruction of objects and humans](https://mlsanity.com/paper/2408.08086/) (2024)
- [SmallPlan: Leverage Small Language Models for Sequential Path Planning with Simulation-Powered, LLM-Guided Distillation](https://mlsanity.com/paper/2505.00831/) (2025)
- [studentSplat: Your Student Model Learns Single-view 3D Gaussian Splatting](https://mlsanity.com/paper/2601.11772/) (2026)
- [ToW3D: Consistency-aware Interactive Point-based Mesh Editing on GANs](https://mlsanity.com/paper/2609.26078/) (2026)

## Abstract (arXiv)

We present a framework for learning 3D object shapes and dense cross-object 3D correspondences from just an unaligned category-specific image collection. The 3D shapes are generated implicitly as deformations to a category-specific signed distance field and are learned in an unsupervised manner solely from unaligned image collections and their poses without any 3D supervision. Generally, image collections on the internet contain several intra-category geometric and topological variations, for example, different chairs can have different topologies, which makes the task of joint shape and correspondence estimation much more challenging. Because of this, prior works either focus on learning each 3D object shape individually without modeling cross-instance correspondences or perform joint shape and correspondence estimation on categories with minimal intra-category topological variations. We overcome these restrictions by learning a topologically-aware implicit deformation field that maps a 3D point in the object space to a higher dimensional point in the category-specific canonical space. At inference time, given a single image, we reconstruct the underlying 3D shape by first implicitly deforming each 3D point in the object space to the learned category-specific canonical space using the topologically-aware deformation field and then reconstructing the 3D shape as a canonical signed distance field. Both canonical shape and deformation field are learned end-to-end in an inverse-graphics fashion using a learned recurrent ray marcher (SRN) as a differentiable rendering module. Our approach, dubbed TARS, achieves state-of-the-art reconstruction fidelity on several datasets: ShapeNet, Pascal3D+, CUB, and Pix3D chairs. Result videos and code at https://shivamduggal4.github.io/tars-3D/