# Watching Swarm Dynamics from Above: A Framework for Advanced Object Tracking in Drone Videos

arXiv: https://arxiv.org/abs/2406.07680 · HTML: https://mlsanity.com/paper/2406.07680/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 60 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Decoupling Features in Hierarchical Propagation for Video Object Segmentation](https://mlsanity.com/paper/2210.09782/) (2022)
- [Follow Anything: Open-set detection, tracking, and following in real-time](https://mlsanity.com/paper/2308.05737/) (2023)
- [SiamMask: A Framework for Fast Online Object Tracking and Segmentation](https://mlsanity.com/paper/2207.02088/) (2022)
- [Particle Swarm Optimisation for Evolving Deep Neural Networks for Image Classification by Evolving and Stacking Transferable Blocks](https://mlsanity.com/paper/1907.12659/) (2019)
- [The Right Spin: Learning Object Motion from Rotation-Compensated Flow Fields](https://mlsanity.com/paper/2203.00115/) (2022)
- [Individuality in Swarm Robots with the Case Study of Kilobots: Noise, Bug, or Feature?](https://mlsanity.com/paper/2305.16063/) (2023)
- [Decentralized traffic management of autonomous drones](https://mlsanity.com/paper/2312.11207/) (2023)

## Built on by (most cited first, 2 of 2)

- [Simultaneous inference of environmental and interaction forces in collective dynamics](https://mlsanity.com/paper/2608.25181/) (2026)
- [Zero-Shot Multi-Animal Tracking in the Wild](https://mlsanity.com/paper/2511.02591/) (2025)

## Abstract (arXiv)

Easily accessible sensors, like drones with diverse onboard sensors, have greatly expanded studying animal behavior in natural environments. Yet, analyzing vast, unlabeled video data, often spanning hours, remains a challenge for machine learning, especially in computer vision. Existing approaches often analyze only a few frames. Our focus is on long-term animal behavior analysis. To address this challenge, we utilize classical probabilistic methods for state estimation, such as particle filtering. By incorporating recent advancements in semantic object segmentation, we enable continuous tracking of rapidly evolving object formations, even in scenarios with limited data availability. Particle filters offer a provably optimal algorithmic structure for recursively adding new incoming information. We propose a novel approach for tracking schools of fish in the open ocean from drone videos. Our framework not only performs classical object tracking in 2D, instead it tracks the position and spatial expansion of the fish school in world coordinates by fusing video data and the drone's on board sensor information (GPS and IMU). The presented framework for the first time allows researchers to study collective behavior of fish schools in its natural social and environmental context in a non-invasive and scalable way.