# 3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking

arXiv: https://arxiv.org/abs/2308.15316 · HTML: https://mlsanity.com/paper/2308.15316/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 105 references that are in the map.
- First posted on arXiv 29 Aug 2023.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Learnable Triangulation of Human Pose](https://mlsanity.com/paper/1905.05754/) (2019)
- [MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking](https://mlsanity.com/paper/2010.07548/) (2020)
- [Creatures great and SMAL: Recovering the shape and motion of animals from video](https://mlsanity.com/paper/1811.05804/) (2018)
- [Deep Cosine Metric Learning for Person Re-Identification](https://mlsanity.com/paper/1812.00442/) (2018)
- [XMem++: Production-level Video Segmentation From Few Annotated Frames](https://mlsanity.com/paper/2307.15958/) (2023)
- [3D Bird Reconstruction: a Dataset, Model, and Shape Recovery from a Single View](https://mlsanity.com/paper/2008.06133/) (2020)
- [AnimalTrack: A Benchmark for Multi-Animal Tracking in the Wild](https://mlsanity.com/paper/2205.00158/) (2022)
- [AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild](https://mlsanity.com/paper/2103.13282/) (2021)
- [3D-ZeF: A 3D Zebrafish Tracking Benchmark Dataset](https://mlsanity.com/paper/2006.08466/) (2020)
- [Joint Object and Part Segmentation using Deep Learned Potentials](https://mlsanity.com/paper/1505.00276/) (2015)
- [Transferring Dense Pose to Proximal Animal Classes](https://mlsanity.com/paper/2003.00080/) (2020)
- [Semantic Part Segmentation using Compositional Model combining Shape and Appearance](https://mlsanity.com/paper/1412.6124/) (2014)
- [3D-POP -- An automated annotation approach to facilitate markerless 2D-3D tracking of freely moving birds with marker-based motion capture](https://mlsanity.com/paper/2303.13174/) (2023)
- [Measuring behavior across scales](https://mlsanity.com/paper/1712.05784/) (2017)
- [BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos](https://mlsanity.com/paper/2212.07401/) (2022)

## Built on by (most cited first, 12 of 12)

- [BuckTales : A multi-UAV dataset for multi-object tracking and re-identification of wild antelopes](https://mlsanity.com/paper/2411.06896/) (2024)
- [SHIC: Shape-Image Correspondences with no Keypoint Supervision](https://mlsanity.com/paper/2407.18907/) (2024)
- [The SA-FARI Dataset: Segment Anything in Footage of Animals for Recognition and Identification](https://mlsanity.com/paper/2511.15622/) (2025)
- [DCHM: Depth-Consistent Human Modeling for Multiview Detection](https://mlsanity.com/paper/2507.14505/) (2025)
- [Context-Aware Outlier Rejection for Robust Multi-View 3D Tracking of Similar Small Birds in An Outdoor Aviary](https://mlsanity.com/paper/2412.16511/) (2024)
- [Towards Application-Specific Evaluation of Vision Models: Case Studies in Ecology and Biology](https://mlsanity.com/paper/2505.02825/) (2025)
- [Neural Texture Puppeteer: A Framework for Neural Geometry and Texture Rendering of Articulated Shapes, Enabling Re-Identification at Interactive Speed](https://mlsanity.com/paper/2311.17109/) (2023)
- [Towards Texture- And Shape-Independent 3D Keypoint Estimation in Birds](https://mlsanity.com/paper/2505.16633/) (2025)
- [SteerPose: Simultaneous Extrinsic Camera Calibration and Matching from Articulation](https://mlsanity.com/paper/2506.01691/) (2025)
- [Robust Multi-view Camera Calibration from Dense Matches](https://mlsanity.com/paper/2512.15608/) (2025)
- [CHIRP dataset: towards long-term, individual-level, behavioral monitoring of bird populations in the wild](https://mlsanity.com/paper/2603.25524/) (2026)
- [Promptable Animal Pose Tracking Across Species](https://mlsanity.com/paper/2608.04995/) (2026)

## Abstract (arXiv)

Markerless methods for animal posture tracking have been rapidly developing recently, but frameworks and benchmarks for tracking large animal groups in 3D are still lacking. To overcome this gap in the literature, we present 3D-MuPPET, a framework to estimate and track 3D poses of up to 10 pigeons at interactive speed using multiple camera views. We train a pose estimator to infer 2D keypoints and bounding boxes of multiple pigeons, then triangulate the keypoints to 3D. For identity matching of individuals in all views, we first dynamically match 2D detections to global identities in the first frame, then use a 2D tracker to maintain IDs across views in subsequent frames. We achieve comparable accuracy to a state of the art 3D pose estimator in terms of median error and Percentage of Correct Keypoints. Additionally, we benchmark the inference speed of 3D-MuPPET, with up to 9.45 fps in 2D and 1.89 fps in 3D, and perform quantitative tracking evaluation, which yields encouraging results. Finally, we showcase two novel applications for 3D-MuPPET. First, we train a model with data of single pigeons and achieve comparable results in 2D and 3D posture estimation for up to 5 pigeons. Second, we show that 3D-MuPPET also works in outdoors without additional annotations from natural environments. Both use cases simplify the domain shift to new species and environments, largely reducing annotation effort needed for 3D posture tracking. To the best of our knowledge we are the first to present a framework for 2D/3D animal posture and trajectory tracking that works in both indoor and outdoor environments for up to 10 individuals. We hope that the framework can open up new opportunities in studying animal collective behaviour and encourages further developments in 3D multi-animal posture tracking.