# MedShapeNet -- A Large-Scale Dataset of 3D Medical Shapes for Computer Vision

arXiv: https://arxiv.org/abs/2308.16139 · HTML: https://mlsanity.com/paper/2308.16139/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 77 of its 215 references that are in the map.
- First posted on arXiv 30 Aug 2023.

## Builds on (most cited first, 30 of 77)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation](https://mlsanity.com/paper/2212.04493/) (2022)
- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [Recent Trends in Deep Learning Based Natural Language Processing](https://mlsanity.com/paper/1708.02709/) (2017)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1706.01307/) (2017)
- [Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis](https://mlsanity.com/paper/1612.00101/) (2016)
- [The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge](https://mlsanity.com/paper/1912.01054/) (2019)
- [A2D2: Audi Autonomous Driving Dataset](https://mlsanity.com/paper/2004.06320/) (2020)
- [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images](https://mlsanity.com/paper/2001.09193/) (2020)
- [Deep Geometric Prior for Surface Reconstruction](https://mlsanity.com/paper/1811.10943/) (2018)
- [ShapeFormer: Transformer-based Shape Completion via Sparse Representation](https://mlsanity.com/paper/2201.10326/) (2022)
- [ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset](https://mlsanity.com/paper/2206.06694/) (2022)
- [CelebV-Text: A Large-Scale Facial Text-Video Dataset](https://mlsanity.com/paper/2303.14717/) (2023)
- [High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference](https://mlsanity.com/paper/1709.07599/) (2017)

## Built on by (most cited first, 30 of 32)

- [CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answering](https://mlsanity.com/paper/2505.16229/) (2025)
- [How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?](https://mlsanity.com/paper/2501.11253/) (2025)
- [SADIR: Shape-Aware Diffusion Models for 3D Image Reconstruction](https://mlsanity.com/paper/2309.03335/) (2023)
- [Expectation-Maximization as the Engine of Scalable Medical Intelligence](https://mlsanity.com/paper/2501.03410/) (2025)
- [Anatomy Completor: A Multi-class Completion Framework for 3D Anatomy Reconstruction](https://mlsanity.com/paper/2309.04956/) (2023)
- [Label Critic: Design Data Before Models](https://mlsanity.com/paper/2411.02753/) (2024)
- [Non-Rigid 3D Shape Correspondences: From Foundations to Open Challenges and Opportunities](https://mlsanity.com/paper/2604.01274/) (2026)
- [See More, Change Less: Anatomy-Aware Diffusion for Contrast Enhancement](https://mlsanity.com/paper/2512.07251/) (2025)
- [GAInS: Gradient Anomaly-aware Biomedical Instance Segmentation](https://mlsanity.com/paper/2409.13988/) (2024)
- [GERA: Geometric Embedding for Efficient Point Registration Analysis](https://mlsanity.com/paper/2410.00589/) (2024)
- [Hierarchical Feature Learning for Medical Point Clouds via State Space Model](https://mlsanity.com/paper/2504.13015/) (2025)
- [A Survey of Medical Point Cloud Shape Learning: Registration, Reconstruction and Variation](https://mlsanity.com/paper/2508.03057/) (2025)
- [From 2D to 3D, Deep Learning-based Shape Reconstruction in Magnetic Resonance Imaging: A Review](https://mlsanity.com/paper/2510.01296/) (2025)
- [GenMed: A Pairwise Generative Reformulation of Medical Diagnostic Tasks](https://mlsanity.com/paper/2605.10645/) (2026)
- [Thermodynamics-informed graph neural networks for real-time simulation of digital human twins](https://mlsanity.com/paper/2412.12034/) (2024)
- [Boosting 3D Liver Shape Datasets with Diffusion Models and Implicit Neural Representations](https://mlsanity.com/paper/2504.19402/) (2025)
- [Memorization in 3D Shape Generation: An Empirical Study](https://mlsanity.com/paper/2512.23628/) (2025)
- [High-Fidelity Medical Shape Generation via Skeletal Latent Diffusion](https://mlsanity.com/paper/2603.07504/) (2026)
- [Mean-field Chaos Diffusion Models](https://mlsanity.com/paper/2406.05396/) (2024)
- [UniRiT: Towards Few-Shot Non-Rigid Point Cloud Registration](https://mlsanity.com/paper/2410.22909/) (2024)
- [Med-PU: Point Cloud Upsampling for High-Fidelity 3D Medical Shape Reconstruction](https://mlsanity.com/paper/2501.16716/) (2025)
- [Shap-MeD](https://mlsanity.com/paper/2503.15562/) (2025)
- [Template-Guided Reconstruction of Pulmonary Segments with Neural Implicit Functions](https://mlsanity.com/paper/2505.08919/) (2025)
- [NFR: Neural Feature-Guided Non-Rigid Shape Registration](https://mlsanity.com/paper/2505.22445/) (2025)
- [ShapeKit](https://mlsanity.com/paper/2506.24003/) (2025)
- [Benchmark-Ready 3D Anatomical Shape Classification](https://mlsanity.com/paper/2511.01613/) (2025)
- [Leveraging a Statistical Shape Model for Efficient Generation of Annotated Training Data: A Case Study on Liver Landmarks Segmentation](https://mlsanity.com/paper/2603.13969/) (2026)
- [Training-free Detection and 6D Pose Estimation of Unseen Surgical Instruments](https://mlsanity.com/paper/2603.25228/) (2026)
- [Structured Analytic Coherent Point Drift for Non-Rigid Point Set Registration](https://mlsanity.com/paper/2605.00934/) (2026)
- [Observation-Conditioned Latent Energy Priors for Sparse Implicit Neural Shape Completion](https://mlsanity.com/paper/2609.03694/) (2026)

## Abstract (arXiv)

Prior to the deep learning era, shape was commonly used to describe the objects. Nowadays, state-of-the-art (SOTA) algorithms in medical imaging are predominantly diverging from computer vision, where voxel grids, meshes, point clouds, and implicit surface models are used. This is seen from numerous shape-related publications in premier vision conferences as well as the growing popularity of ShapeNet (about 51,300 models) and Princeton ModelNet (127,915 models). For the medical domain, we present a large collection of anatomical shapes (e.g., bones, organs, vessels) and 3D models of surgical instrument, called MedShapeNet, created to facilitate the translation of data-driven vision algorithms to medical applications and to adapt SOTA vision algorithms to medical problems. As a unique feature, we directly model the majority of shapes on the imaging data of real patients. As of today, MedShapeNet includes 23 dataset with more than 100,000 shapes that are paired with annotations (ground truth). Our data is freely accessible via a web interface and a Python application programming interface (API) and can be used for discriminative, reconstructive, and variational benchmarks as well as various applications in virtual, augmented, or mixed reality, and 3D printing. Exemplary, we present use cases in the fields of classification of brain tumors, facial and skull reconstructions, multi-class anatomy completion, education, and 3D printing. In future, we will extend the data and improve the interfaces. The project pages are: https://medshapenet.ikim.nrw/ and https://github.com/Jianningli/medshapenet-feedback