# TrackFish3D: Self-Supervised 3D Tracking of Schooling Fish from Multi-view Videos

arXiv: https://arxiv.org/abs/2609.38347 · HTML: https://mlsanity.com/paper/2609.38347/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 61 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 29 of 29)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [MeMOT: Multi-Object Tracking with Memory](https://mlsanity.com/paper/2203.16761/) (2022)
- [Multiple Object Tracking as ID Prediction](https://mlsanity.com/paper/2403.16848/) (2024)
- [Abnormal Event Detection in Videos using Spatiotemporal Autoencoder](https://mlsanity.com/paper/1701.01546/) (2017)
- [Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models](https://mlsanity.com/paper/2606.03748/) (2026)
- [Anomaly Detection with Generative Adversarial Networks for Multivariate Time Series](https://mlsanity.com/paper/1809.04758/) (2018)
- [LMGP: Lifted Multicut Meets Geometry Projections for Multi-Camera Multi-Object Tracking](https://mlsanity.com/paper/2111.11892/) (2021)
- [ADA-Track++: End-to-End Multi-Camera 3D Multi-Object Tracking with Alternating Detection and Association](https://mlsanity.com/paper/2405.08909/) (2024)
- [DyGLIP: A Dynamic Graph Model with Link Prediction for Accurate Multi-Camera Multiple Object Tracking](https://mlsanity.com/paper/2106.06856/) (2021)
- [3D-ZeF: A 3D Zebrafish Tracking Benchmark Dataset](https://mlsanity.com/paper/2006.08466/) (2020)
- [ReST: A Reconfigurable Spatial-Temporal Graph Model for Multi-Camera Multi-Object Tracking](https://mlsanity.com/paper/2308.13229/) (2023)
- [Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking](https://mlsanity.com/paper/2410.01806/) (2024)
- [3D-POP -- An automated annotation approach to facilitate markerless 2D-3D tracking of freely moving birds with marker-based motion capture](https://mlsanity.com/paper/2303.13174/) (2023)
- [MessyTable: Instance Association in Multiple Camera Views](https://mlsanity.com/paper/2007.14878/) (2020)
- [Self-supervised Multi-view Person Association and Its Applications](https://mlsanity.com/paper/1805.08717/) (2018)
- [Collective behavior from surprise minimization](https://mlsanity.com/paper/2307.14804/) (2023)
- [MCTR: Multi Camera Tracking Transformer](https://mlsanity.com/paper/2408.13243/) (2024)
- [Learning from Synchronization: Self-Supervised Uncalibrated Multi-View Person Association in Challenging Scenes](https://mlsanity.com/paper/2503.13739/) (2025)
- [CBIL: Collective Behavior Imitation Learning for Fish from Real Videos](https://mlsanity.com/paper/2504.00234/) (2025)
- [idtracker.ai: Tracking all individuals in large collectives of unmarked animals](https://mlsanity.com/paper/1803.04351/) (2018)

## Abstract (arXiv)

Quantifying collective fish behavior requires accurate trajectories, yet multi-view 3D tracking remains challenging due to frequent occlusions, visually similar individuals, and the long-standing scarcity of identity annotations. We present TrackFish3D, a geometry-driven self-supervised framework for dense multi-camera 3D tracking of schooling fish. Instead of relying on appearance-based re-identification or manually annotated identities, TrackFish3D turns calibrated multi-view geometry into supervision: triangulation and reprojection consistency provide pseudo-associations, while a geometric encoder and global association transformer learn all-to-all cross-view correspondence within each frame. To make these associations identity-aware, TrackFish3D introduces a self-supervised contrastive objective that separates co-visible individuals in the embedding space, together with a temporal predictor that preserves identities and bridges short occlusions across frames. The resulting model is trained once on unlabeled footage and applied directly to unseen test videos, requiring no cross-view identity labels, temporal annotations, 3D ground truth, appearance features, or test-time optimization. On our benchmark, TrackFish3D improves 3D Multi-Object Tracking Accuracy from 87.7% for the strongest baseline to 95.8%. On the 3D-ZeF zebrafish benchmark, it achieves 81.1% MOTA, compared with 77.4% for the best geometric baseline. TrackFish3D also generalizes beyond fish, achieving strong results on real-world bird tracking.