# DeepSSM: A Blueprint for Image-to-Shape Deep Learning Models

arXiv: https://arxiv.org/abs/2110.07152 · HTML: https://mlsanity.com/paper/2110.07152/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 66 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Learning a Predictable and Generative Vector Representation for Objects](https://mlsanity.com/paper/1603.08637/) (2016)
- [An Unsupervised Learning Model for Deformable Medical Image Registration](https://mlsanity.com/paper/1802.02604/) (2018)
- [Quicksilver: Fast Predictive Image Registration - a Deep Learning Approach](https://mlsanity.com/paper/1703.10908/) (2017)
- [Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation](https://mlsanity.com/paper/1705.08302/) (2017)
- [Anatomical Priors in Convolutional Networks for Unsupervised Biomedical Segmentation](https://mlsanity.com/paper/1903.03148/) (2019)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling](https://mlsanity.com/paper/1505.07293/) (2015)
- [DeepSSM: A Deep Learning Framework for Statistical Shape Modeling from Raw Images](https://mlsanity.com/paper/1810.00111/) (2018)
- [Probabilistic 3D surface reconstruction from sparse MRI information](https://mlsanity.com/paper/2010.02041/) (2020)
- [Benchmarking off-the-shelf statistical shape modeling tools in clinical applications](https://mlsanity.com/paper/2009.02878/) (2020)
- [Temporal HeartNet: Towards Human-Level Automatic Analysis of Fetal Cardiac Screening Video](https://mlsanity.com/paper/1707.00665/) (2017)
- [Uncertainty Quantification in CNN-Based Surface Prediction Using Shape Priors](https://mlsanity.com/paper/1807.11272/) (2018)
- [A Cooperative Autoencoder for Population-Based Regularization of CNN Image Registration](https://mlsanity.com/paper/1908.05825/) (2019)
- [Deep Learning for End-to-End Atrial Fibrillation Recurrence Estimation](https://mlsanity.com/paper/1810.00475/) (2018)
- [On the Evaluation and Validation of Off-the-shelf Statistical Shape Modeling Tools: A Clinical Application](https://mlsanity.com/paper/1810.03987/) (2018)
- [Unsupervised Shape Normality Metric for Severity Quantification](https://mlsanity.com/paper/2007.09307/) (2020)
- [Deep Shape Analysis on Abdominal Organs for Diabetes Prediction](https://mlsanity.com/paper/1808.01946/) (2018)

## Built on by (most cited first, 25 of 25)

- [Improving anatomical plausibility in medical image segmentation via hybrid graph neural networks: applications to chest x-ray analysis](https://mlsanity.com/paper/2203.10977/) (2022)
- [From Images to Probabilistic Anatomical Shapes: A Deep Variational Bottleneck Approach](https://mlsanity.com/paper/2205.06862/) (2022)
- [Point2SSM: Learning Morphological Variations of Anatomies from Point Cloud](https://mlsanity.com/paper/2305.14486/) (2023)
- [S3M: Scalable Statistical Shape Modeling through Unsupervised Correspondences](https://mlsanity.com/paper/2304.07515/) (2023)
- [Image2SSM: Reimagining Statistical Shape Models from Images with Radial Basis Functions](https://mlsanity.com/paper/2305.11946/) (2023)
- [Point2SSM++: Self-Supervised Learning of Anatomical Shape Models from Point Clouds](https://mlsanity.com/paper/2405.09707/) (2024)
- [ADASSM: Adversarial Data Augmentation in Statistical Shape Models From Images](https://mlsanity.com/paper/2307.03273/) (2023)
- [Weakly Supervised Bayesian Shape Modeling from Unsegmented Medical Images](https://mlsanity.com/paper/2405.09697/) (2024)
- [SLoRD: Structural Low-Rank Descriptors for Shape Consistency in Vertebrae Segmentation](https://mlsanity.com/paper/2407.08555/) (2024)
- [From Pixels to Polygons: A Survey of Deep Learning Approaches for Medical Image-to-Mesh Reconstruction](https://mlsanity.com/paper/2505.03599/) (2025)
- [SCorP: Statistics-Informed Dense Correspondence Prediction Directly from Unsegmented Medical Images](https://mlsanity.com/paper/2404.17967/) (2024)
- [Learning Population-level Shape Statistics and Anatomy Segmentation From Images: A Joint Deep Learning Model](https://mlsanity.com/paper/2201.03481/) (2022)
- [Progressive DeepSSM: Training Methodology for Image-To-Shape Deep Models](https://mlsanity.com/paper/2310.01529/) (2023)
- [MASSM: An End-to-End Deep Learning Framework for Multi-Anatomy Statistical Shape Modeling Directly From Images](https://mlsanity.com/paper/2403.11008/) (2024)
- [Mesh2SSM++: A Probabilistic Framework for Unsupervised Learning of Statistical Shape Model of Anatomies from Surface Meshes](https://mlsanity.com/paper/2502.07145/) (2025)
- [A Survey of Medical Point Cloud Shape Learning: Registration, Reconstruction and Variation](https://mlsanity.com/paper/2508.03057/) (2025)
- [Cortex-Grounded Diffusion Models for Brain Image Generation](https://mlsanity.com/paper/2601.19498/) (2026)
- [Mask-HybridGNet: Graph-based segmentation with emergent anatomical correspondence from pixel-level supervision](https://mlsanity.com/paper/2602.21179/) (2026)
- [Probabilistic 3D Correspondence Prediction from Sparse Unsegmented Images](https://mlsanity.com/paper/2407.01931/) (2024)
- [On the Viability of Semi-Supervised Segmentation Methods for Statistical Shape Modeling](https://mlsanity.com/paper/2407.15260/) (2024)
- [Implicit Shape-Prior for Few-Shot Assisted 3D Segmentation](https://mlsanity.com/paper/2509.08580/) (2025)
- [MorphoFlow: Sparse-Supervised Generative Shape Modeling with Adaptive Latent Relevance](https://mlsanity.com/paper/2604.11636/) (2026)
- [Anatomy-Aware 3D Mesh Refinement of Pericardium Segmentations on Computed Tomography](https://mlsanity.com/paper/2607.19210/) (2026)
- [TRACE: Artifact-Robust Statistical Shape Modeling from Imperfect Surface Scans - A Case Study in Craniosynostosis 3D Photography](https://mlsanity.com/paper/2608.22131/) (2026)
- [A Joint 2D-3D Statistical Shape Model for Orthopedic Reconstruction](https://mlsanity.com/paper/2609.09010/) (2026)

## Abstract (arXiv)

Statistical shape modeling (SSM) characterizes anatomical variations in a population of shapes generated from medical images. SSM requires consistent shape representation across samples in shape cohort. Establishing this representation entails a processing pipeline that includes anatomy segmentation, re-sampling, registration, and non-linear optimization. These shape representations are then used to extract low-dimensional shape descriptors that facilitate subsequent analyses in different applications. However, the current process of obtaining these shape descriptors from imaging data relies on human and computational resources, requiring domain expertise for segmenting anatomies of interest. Moreover, this same taxing pipeline needs to be repeated to infer shape descriptors for new image data using a pre-trained/existing shape model. Here, we propose DeepSSM, a deep learning-based framework for learning the functional mapping from images to low-dimensional shape descriptors and their associated shape representations, thereby inferring statistical representation of anatomy directly from 3D images. Once trained using an existing shape model, DeepSSM circumvents the heavy and manual pre-processing and segmentation and significantly improves the computational time, making it a viable solution for fully end-to-end SSM applications. In addition, we introduce a model-based data-augmentation strategy to address data scarcity. Finally, this paper presents and analyzes two different architectural variants of DeepSSM with different loss functions using three medical datasets and their downstream clinical application. Experiments showcase that DeepSSM performs comparably or better to the state-of-the-art SSM both quantitatively and on application-driven downstream tasks. Therefore, DeepSSM aims to provide a comprehensive blueprint for deep learning-based image-to-shape models.