# VoxelEmbed: 3D Instance Segmentation and Tracking with Voxel Embedding based Deep Learning

arXiv: https://arxiv.org/abs/2106.11480 · HTML: https://mlsanity.com/paper/2106.11480/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 34 references that are in the map.
- First posted on arXiv 22 Jun 2021.

## Builds on (most cited first, 5 of 5)

- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Delving Deeper into Convolutional Networks for Learning Video Representations](https://mlsanity.com/paper/1511.06432/) (2015)
- [Accurate Weakly-Supervised Deep Lesion Segmentation using Large-Scale Clinical Annotations: Slice-Propagated 3D Mask Generation from 2D RECIST](https://mlsanity.com/paper/1807.01172/) (2018)
- [Faster Mean-shift: GPU-accelerated clustering for cosine embedding-based cell segmentation and tracking](https://mlsanity.com/paper/2007.14283/) (2020)
- [Automatic coding of students' writing via Contrastive Representation Learning in the Wasserstein space](https://mlsanity.com/paper/2011.13384/) (2020)

## Built on by (most cited first, 5 of 5)

- [EmbedTrack -- Simultaneous Cell Segmentation and Tracking Through Learning Offsets and Clustering Bandwidths](https://mlsanity.com/paper/2204.10713/) (2022)
- [Collaborative Training of Medical Artificial Intelligence Models with non-uniform Labels](https://mlsanity.com/paper/2211.13606/) (2022)
- [ParticleSeg3D: A Scalable Out-of-the-Box Deep Learning Segmentation Solution for Individual Particle Characterization from Micro CT Images in Mineral Processing and Recycling](https://mlsanity.com/paper/2301.13319/) (2023)
- [Deep Generative Models on 3D Representations: A Survey](https://mlsanity.com/paper/2210.15663/) (2022)
- [QueryGaussian: Scalable and Training-Free Open-Vocabulary 3D Instance Retrieval](https://mlsanity.com/paper/2606.19733/) (2026)

## Abstract (arXiv)

Recent advances in bioimaging have provided scientists a superior high spatial-temporal resolution to observe dynamics of living cells as 3D volumetric videos. Unfortunately, the 3D biomedical video analysis is lagging, impeded by resource insensitive human curation using off-the-shelf 3D analytic tools. Herein, biologists often need to discard a considerable amount of rich 3D spatial information by compromising on 2D analysis via maximum intensity projection. Recently, pixel embedding-based cell instance segmentation and tracking provided a neat and generalizable computing paradigm for understanding cellular dynamics. In this work, we propose a novel spatial-temporal voxel-embedding (VoxelEmbed) based learning method to perform simultaneous cell instance segmenting and tracking on 3D volumetric video sequences. Our contribution is in four-fold: (1) The proposed voxel embedding generalizes the pixel embedding with 3D context information; (2) Present a simple multi-stream learning approach that allows effective spatial-temporal embedding; (3) Accomplished an end-to-end framework for one-stage 3D cell instance segmentation and tracking without heavy parameter tuning; (4) The proposed 3D quantification is memory efficient via a single GPU with 12 GB memory. We evaluate our VoxelEmbed method on four 3D datasets (with different cell types) from the ISBI Cell Tracking Challenge. The proposed VoxelEmbed method achieved consistent superior overall performance (OP) on two densely annotated datasets. The performance is also competitive on two sparsely annotated cohorts with 20.6% and 2% of data-set having segmentation annotations. The results demonstrate that the VoxelEmbed method is a generalizable and memory-efficient solution.