# How to Track and Segment Fish without Human Annotations: A Self-Supervised Deep Learning Approach

arXiv: https://arxiv.org/abs/2208.10662 · HTML: https://mlsanity.com/paper/2208.10662/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 95 references that are in the map.
- First posted on arXiv 23 Aug 2022.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Deep Learning in Video Multi-Object Tracking: A Survey](https://mlsanity.com/paper/1907.12740/) (2019)
- [A Survey of Deep Learning-based Object Detection](https://mlsanity.com/paper/1907.09408/) (2019)
- [TensorMask: A Foundation for Dense Object Segmentation](https://mlsanity.com/paper/1903.12174/) (2019)
- [Video Object Segmentation Without Temporal Information](https://mlsanity.com/paper/1709.06031/) (2017)
- [SegFlow: Joint Learning for Video Object Segmentation and Optical Flow](https://mlsanity.com/paper/1709.06750/) (2017)
- [A Realistic Fish-Habitat Dataset to Evaluate Algorithms for Underwater Visual Analysis](https://mlsanity.com/paper/2008.12603/) (2020)
- [Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlook](https://mlsanity.com/paper/2112.02838/) (2021)
- [The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos](https://mlsanity.com/paper/2111.06394/) (2021)
- [DeepUSPS: Deep Robust Unsupervised Saliency Prediction With Self-Supervision](https://mlsanity.com/paper/1909.13055/) (2019)
- [Every Frame Counts: Joint Learning of Video Segmentation and Optical Flow](https://mlsanity.com/paper/1911.12739/) (2019)
- [Background Subtraction in Real Applications: Challenges, Current Models and Future Directions](https://mlsanity.com/paper/1901.03577/) (2019)
- [Deep Neural Network Concepts for Background Subtraction: A Systematic Review and Comparative Evaluation](https://mlsanity.com/paper/1811.05255/) (2018)
- [Dense Unsupervised Learning for Video Segmentation](https://mlsanity.com/paper/2111.06265/) (2021)
- [Contrastive Transformation for Self-supervised Correspondence Learning](https://mlsanity.com/paper/2012.05057/) (2020)
- [Computer Vision and Deep Learning for Fish Classification in Underwater Habitats: A Survey](https://mlsanity.com/paper/2203.06951/) (2022)
- [Moving Objects Detection with a Moving Camera: A Comprehensive Review](https://mlsanity.com/paper/2001.05238/) (2020)
- [SiamCorners: Siamese Corner Networks for Visual Tracking](https://mlsanity.com/paper/2104.07303/) (2021)
- [Underwater Fish Detection with Weak Multi-Domain Supervision](https://mlsanity.com/paper/1905.10708/) (2019)
- [Automatic Weight Estimation of Harvested Fish from Images](https://mlsanity.com/paper/1909.02710/) (2019)

## Built on by (most cited first, 1 of 1)

- [Supervised Contrastive Frame Aggregation for Video Representation Learning](https://mlsanity.com/paper/2512.12549/) (2025)

## Abstract (arXiv)

Tracking fish movements and sizes of fish is crucial to understanding their ecology and behaviour. Knowing where fish migrate, how they interact with their environment, and how their size affects their behaviour can help ecologists develop more effective conservation and management strategies to protect fish populations and their habitats. Deep learning is a promising tool to analyze fish ecology from underwater videos. However, training deep neural networks (DNNs) for fish tracking and segmentation requires high-quality labels, which are expensive to obtain. We propose an alternative unsupervised approach that relies on spatial and temporal variations in video data to generate noisy pseudo-ground-truth labels. We train a multitask DNN using these pseudo-labels. Our framework consists of three stages: (1) an optical flow model generates the pseudo labels using spatial and temporal consistency between frames, (2) a self-supervised model refines the pseudo-labels incrementally, and (3) a segmentation network uses the refined labels for training. Consequently, we perform extensive experiments to validate our method on three public underwater video datasets and demonstrate its effectiveness for video annotation and segmentation. We also evaluate its robustness to different imaging conditions and discuss its limitations.