# SENDD: Sparse Efficient Neural Depth and Deformation for Tissue Tracking

arXiv: https://arxiv.org/abs/2305.06477 · HTML: https://mlsanity.com/paper/2305.06477/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 19 of 19)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [GA-Net: Guided Aggregation Net for End-to-end Stereo Matching](https://mlsanity.com/paper/1904.06587/) (2019)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [RAFT-3D: Scene Flow using Rigid-Motion Embeddings](https://mlsanity.com/paper/2012.00726/) (2020)
- [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021)
- [SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics](https://mlsanity.com/paper/1909.05405/) (2019)
- [DefSLAM: Tracking and Mapping of Deforming Scenes from Monocular Sequences](https://mlsanity.com/paper/1908.08918/) (2019)
- [MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing](https://mlsanity.com/paper/1803.02009/) (2018)
- [SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction](https://mlsanity.com/paper/2003.03472/) (2020)
- [SD-DefSLAM: Semi-Direct Monocular SLAM for Deformable and Intracorporeal Scenes](https://mlsanity.com/paper/2010.09409/) (2020)
- [Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking](https://mlsanity.com/paper/2210.16674/) (2022)
- [Direct and Sparse Deformable Tracking](https://mlsanity.com/paper/2109.07370/) (2021)
- [ColDE: A Depth Estimation Framework for Colonoscopy Reconstruction](https://mlsanity.com/paper/2111.10371/) (2021)
- [SuperPoint features in endoscopy](https://mlsanity.com/paper/2203.04302/) (2022)
- [BDIS: Bayesian Dense Inverse Searching Method for Real-Time Stereo Surgical Image Matching](https://mlsanity.com/paper/2205.03133/) (2022)

## Built on by (most cited first, 14 of 14)

- [Tracking and Mapping in Medical Computer Vision: A Review](https://mlsanity.com/paper/2310.11475/) (2023)
- [SurgPose: a Dataset for Articulated Robotic Surgical Tool Pose Estimation and Tracking](https://mlsanity.com/paper/2502.11534/) (2025)
- [Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping](https://mlsanity.com/paper/2309.16782/) (2023)
- [Point Tracking in Surgery--The 2024 Surgical Tattoos in Infrared (STIR) Challenge](https://mlsanity.com/paper/2503.24306/) (2025)
- [RIDE: Self-Supervised Learning of Rotation-Equivariant Keypoint Detection and Invariant Description for Endoscopy](https://mlsanity.com/paper/2309.09563/) (2023)
- [Tracking Everything in Robotic-Assisted Surgery](https://mlsanity.com/paper/2409.19821/) (2024)
- [LiteTracker: Leveraging Temporal Causality for Accurate Low-latency Tissue Tracking](https://mlsanity.com/paper/2504.09904/) (2025)
- [A-MFST: Adaptive Multi-Flow Sparse Tracker for Real-Time Tissue Tracking Under Occlusion](https://mlsanity.com/paper/2410.19996/) (2024)
- [Endo-TTAP: Robust Endoscopic Tissue Tracking via Multi-Facet Guided Attention and Hybrid Flow-point Supervision](https://mlsanity.com/paper/2503.22394/) (2025)
- [MultiViPerFrOG: A Globally Optimized Multi-Viewpoint Perception Framework for Camera Motion and Tissue Deformation](https://mlsanity.com/paper/2408.04367/) (2024)
- [You Are Your Best Teacher: Semi-Supervised Surgical Point Tracking with Cycle-Consistent Self-Distillation](https://mlsanity.com/paper/2505.05722/) (2025)
- [Surg-InvNeRF: Invertible NeRF for 3D tracking and reconstruction in surgical vision](https://mlsanity.com/paper/2508.09681/) (2025)
- [Bridging Vision and Language for Robust Context-Aware Surgical Point Tracking: The VL-SurgPT Dataset and Benchmark](https://mlsanity.com/paper/2511.12026/) (2025)
- [LapaTrack-3D: 6 DoF pre-operative shape tracking for laparoscopic surgery](https://mlsanity.com/paper/2609.19954/) (2026)

## Abstract (arXiv)

Deformable tracking and real-time estimation of 3D tissue motion is essential to enable automation and image guidance applications in robotically assisted surgery. Our model, Sparse Efficient Neural Depth and Deformation (SENDD), extends prior 2D tracking work to estimate flow in 3D space. SENDD introduces novel contributions of learned detection, and sparse per-point depth and 3D flow estimation, all with less than half a million parameters. SENDD does this by using graph neural networks of sparse keypoint matches to estimate both depth and 3D flow anywhere. We quantify and benchmark SENDD on a comprehensively labelled tissue dataset, and compare it to an equivalent 2D flow model. SENDD performs comparably while enabling applications that 2D flow cannot. SENDD can track points and estimate depth at 10fps on an NVIDIA RTX 4000 for 1280 tracked (query) points and its cost scales linearly with an increasing/decreasing number of points. SENDD enables multiple downstream applications that require estimation of 3D motion in stereo endoscopy.