# Joint Learning of 3D Shape Retrieval and Deformation

arXiv: https://arxiv.org/abs/2101.07889 · HTML: https://mlsanity.com/paper/2101.07889/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 58 references that are in the map.
- First posted on arXiv 19 Jan 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)
- [Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis](https://mlsanity.com/paper/1612.00101/) (2016)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [SDM-NET: Deep Generative Network for Structured Deformable Mesh](https://mlsanity.com/paper/1908.04520/) (2019)
- [Neural Cages for Detail-Preserving 3D Deformations](https://mlsanity.com/paper/1912.06395/) (2019)
- [Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation](https://mlsanity.com/paper/1908.01491/) (2019)
- [3DN: 3D Deformation Network](https://mlsanity.com/paper/1903.03322/) (2019)
- [Scan2Mesh: From Unstructured Range Scans to 3D Meshes](https://mlsanity.com/paper/1811.10464/) (2018)
- [Deformation-Aware 3D Model Embedding and Retrieval](https://mlsanity.com/paper/2004.01228/) (2020)
- [ShapeFlow: Learnable Deformations Among 3D Shapes](https://mlsanity.com/paper/2006.07982/) (2020)
- [DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image](https://mlsanity.com/paper/1708.04672/) (2017)
- [Learning Free-Form Deformations for 3D Object Reconstruction](https://mlsanity.com/paper/1803.10932/) (2018)
- [Joint Embedding of 3D Scan and CAD Objects](https://mlsanity.com/paper/1908.06989/) (2019)
- [CAD-Deform: Deformable Fitting of CAD Models to 3D Scans](https://mlsanity.com/paper/2007.11965/) (2020)
- [Parsing Geometry Using Structure-Aware Shape Templates](https://mlsanity.com/paper/1808.01337/) (2018)
- [DeformSyncNet: Deformation Transfer via Synchronized Shape Deformation Spaces](https://mlsanity.com/paper/2009.01456/) (2020)

## Built on by (most cited first, 30 of 42)

- [CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion](https://mlsanity.com/paper/2305.16283/) (2023)
- [DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Image](https://mlsanity.com/paper/2311.18610/) (2023)
- [Reconstructing editable prismatic CAD from rounded voxel models](https://mlsanity.com/paper/2209.01161/) (2022)
- [Img2CAD: Reverse Engineering 3D CAD Models from Images through VLM-Assisted Conditional Factorization](https://mlsanity.com/paper/2408.01437/) (2024)
- [Neural Template: Topology-aware Reconstruction and Disentangled Generation of 3D Meshes](https://mlsanity.com/paper/2206.04942/) (2022)
- [SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation](https://mlsanity.com/paper/2201.13168/) (2022)
- [CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation](https://mlsanity.com/paper/2311.01447/) (2023)
- [Point Scene Understanding via Disentangled Instance Mesh Reconstruction](https://mlsanity.com/paper/2203.16832/) (2022)
- [MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation](https://mlsanity.com/paper/2408.14899/) (2024)
- [ANISE: Assembly-based Neural Implicit Surface rEconstruction](https://mlsanity.com/paper/2205.13682/) (2022)
- [Geometry in Style: 3D Stylization via Surface Normal Deformation](https://mlsanity.com/paper/2503.23241/) (2025)
- [U-RED: Unsupervised 3D Shape Retrieval and Deformation for Partial Point Clouds](https://mlsanity.com/paper/2308.06383/) (2023)
- [MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervision](https://mlsanity.com/paper/2310.11696/) (2023)
- [UNOPose: Unseen Object Pose Estimation with an Unposed RGB-D Reference Image](https://mlsanity.com/paper/2411.16106/) (2024)
- [SCoDA: Domain Adaptive Shape Completion for Real Scans](https://mlsanity.com/paper/2304.10179/) (2023)
- [Reduced Representation of Deformation Fields for Effective Non-rigid Shape Matching](https://mlsanity.com/paper/2211.14604/) (2022)
- [ShapeMatcher: Self-Supervised Joint Shape Canonicalization, Segmentation, Retrieval and Deformation](https://mlsanity.com/paper/2311.11106/) (2023)
- [Generalizing Single-View 3D Shape Retrieval to Occlusions and Unseen Objects](https://mlsanity.com/paper/2401.00405/) (2023)
- [Intuitive Shape Editing in Latent Space](https://mlsanity.com/paper/2111.12488/) (2021)
- [PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval and Deformation](https://mlsanity.com/paper/2207.11790/) (2022)
- [Structure-Aware 3D VR Sketch to 3D Shape Retrieval](https://mlsanity.com/paper/2209.09043/) (2022)
- [Unsupervised 3D Shape Reconstruction by Part Retrieval and Assembly](https://mlsanity.com/paper/2303.01999/) (2023)
- [Digital Twin Generation from Visual Data: A Survey](https://mlsanity.com/paper/2504.13159/) (2025)
- [KP-RED: Exploiting Semantic Keypoints for Joint 3D Shape Retrieval and Deformation](https://mlsanity.com/paper/2403.10099/) (2024)
- [COM3D: Leveraging Cross-View Correspondence and Cross-Modal Mining for 3D Retrieval](https://mlsanity.com/paper/2405.04103/) (2024)
- [GIVEPose: Gradual Intra-class Variation Elimination for RGB-based Category-Level Object Pose Estimation](https://mlsanity.com/paper/2503.15110/) (2025)
- [NeuralMLS: Geometry-Aware Control Point Deformation](https://mlsanity.com/paper/2201.01873/) (2022)
- [Accurate Instance-Level CAD Model Retrieval in a Large-Scale Database](https://mlsanity.com/paper/2207.01339/) (2022)
- [MXM-CLR: A Unified Framework for Contrastive Learning of Multifold Cross-Modal Representations](https://mlsanity.com/paper/2303.10839/) (2023)
- [Deep Sketch-Based 3D Modeling: A Survey](https://mlsanity.com/paper/2603.03287/) (2026)

## Abstract (arXiv)

We propose a novel technique for producing high-quality 3D models that match a given target object image or scan. Our method is based on retrieving an existing shape from a database of 3D models and then deforming its parts to match the target shape. Unlike previous approaches that independently focus on either shape retrieval or deformation, we propose a joint learning procedure that simultaneously trains the neural deformation module along with the embedding space used by the retrieval module. This enables our network to learn a deformation-aware embedding space, so that retrieved models are more amenable to match the target after an appropriate deformation. In fact, we use the embedding space to guide the shape pairs used to train the deformation module, so that it invests its capacity in learning deformations between meaningful shape pairs. Furthermore, our novel part-aware deformation module can work with inconsistent and diverse part-structures on the source shapes. We demonstrate the benefits of our joint training not only on our novel framework, but also on other state-of-the-art neural deformation modules proposed in recent years. Lastly, we also show that our jointly-trained method outperforms various non-joint baselines.