# 3D shape reconstruction of semi-transparent worms

arXiv: https://arxiv.org/abs/2304.14841 · HTML: https://mlsanity.com/paper/2304.14841/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 71 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose](https://mlsanity.com/paper/1611.07828/) (2016)
- [Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild](https://mlsanity.com/paper/1911.11130/) (2019)
- [A Survey of Structure from Motion](https://mlsanity.com/paper/1701.08493/) (2017)
- [3D Human Pose Estimation = 2D Pose Estimation + Matching](https://mlsanity.com/paper/1612.06524/) (2016)
- [Creatures great and SMAL: Recovering the shape and motion of animals from video](https://mlsanity.com/paper/1811.05804/) (2018)
- [Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop](https://mlsanity.com/paper/2007.11110/) (2020)
- [Recent Advances in Monocular 2D and 3D Human Pose Estimation: A Deep Learning Perspective](https://mlsanity.com/paper/2104.11536/) (2021)
- [RGBD-Dog: Predicting Canine Pose from RGBD Sensors](https://mlsanity.com/paper/2004.07788/) (2020)
- [Learning 3D Deformation of Animals from 2D Images](https://mlsanity.com/paper/1507.07646/) (2015)
- [A MultiPath Network for Object Detection](https://mlsanity.com/paper/1604.02135/) (2016)

## Abstract (arXiv)

3D shape reconstruction typically requires identifying object features or textures in multiple images of a subject. This approach is not viable when the subject is semi-transparent and moving in and out of focus. Here we overcome these challenges by rendering a candidate shape with adaptive blurring and transparency for comparison with the images. We use the microscopic nematode Caenorhabditis elegans as a case study as it freely explores a 3D complex fluid with constantly changing optical properties. We model the slender worm as a 3D curve using an intrinsic parametrisation that naturally admits biologically-informed constraints and regularisation. To account for the changing optics we develop a novel differentiable renderer to construct images from 2D projections and compare against raw images to generate a pixel-wise error to jointly update the curve, camera and renderer parameters using gradient descent. The method is robust to interference such as bubbles and dirt trapped in the fluid, stays consistent through complex sequences of postures, recovers reliable estimates from blurry images and provides a significant improvement on previous attempts to track C. elegans in 3D. Our results demonstrate the potential of direct approaches to shape estimation in complex physical environments in the absence of ground-truth data.