# Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance

arXiv: https://arxiv.org/abs/2608.18602 · HTML: https://mlsanity.com/paper/2608.18602/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Builds on (most cited first, 25 of 25)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [Perception Encoder: The best visual embeddings are not at the output of the network](https://mlsanity.com/paper/2504.13181/) (2025)
- [Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding](https://mlsanity.com/paper/2601.10611/) (2026)
- [PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding](https://mlsanity.com/paper/2504.13180/) (2025)
- [Efficient Pipeline for Camera Trap Image Review](https://mlsanity.com/paper/1907.06772/) (2019)
- [XMem++: Production-level Video Segmentation From Few Annotated Frames](https://mlsanity.com/paper/2307.15958/) (2023)
- [Active Statistical Inference](https://mlsanity.com/paper/2403.03208/) (2024)
- [3D-ZeF: A 3D Zebrafish Tracking Benchmark Dataset](https://mlsanity.com/paper/2006.08466/) (2020)
- [The Caltech Fish Counting Dataset: A Benchmark for Multiple-Object Tracking and Counting](https://mlsanity.com/paper/2207.09295/) (2022)
- [Learning the What and How of Annotation in Video Object Segmentation](https://mlsanity.com/paper/2311.04414/) (2023)
- [Pytorch-Wildlife: A Collaborative Deep Learning Framework for Conservation](https://mlsanity.com/paper/2405.12930/) (2024)
- [BuckTales : A multi-UAV dataset for multi-object tracking and re-identification of wild antelopes](https://mlsanity.com/paper/2411.06896/) (2024)
- [MammAlps: A multi-view video behavior monitoring dataset of wild mammals in the Swiss Alps](https://mlsanity.com/paper/2503.18223/) (2025)
- [Align and Distill: Unifying and Improving Domain Adaptive Object Detection](https://mlsanity.com/paper/2403.12029/) (2024)
- [The SA-FARI Dataset: Segment Anything in Footage of Animals for Recognition and Identification](https://mlsanity.com/paper/2511.15622/) (2025)
- [FBD-SV-2024: Flying Bird Object Detection Dataset in Surveillance Video](https://mlsanity.com/paper/2409.00317/) (2024)
- [WildLive: Near Real-time Visual Wildlife Tracking onboard UAVs](https://mlsanity.com/paper/2504.10165/) (2025)
- [Finding Needles in the Haystack: Transductive Active Labeling in Ecology](https://mlsanity.com/paper/2606.03821/) (2026)
- [No Free Lunch in Annotation either: An objective evaluation of foundation models for streamlining annotation in animal tracking](https://mlsanity.com/paper/2502.03907/) (2025)

## Abstract (arXiv)

Computer vision is increasingly used to automate recognition tasks in large ecological datasets, but more complex tasks such as multi-object tracking continue to pose challenges. As researchers seek to incorporate vision models in ecology workflows, various lines of research have explored how to make imperfect predictions useful through human-in-the-loop processes. We propose a new approach to working with imperfect tracking predictions through an interactive prediction correction workflow taking place as a conversation with a multimodal large language model, which we tailor to a sonar fish tracking dataset as an initial proof of concept. We investigate the performance of the tool, Molmo2Fish, across guided and unguided tasks, correcting its own predicted tracks and external tracks. We find that Molmo2Fish achieves high performance on fish tracking and track correction tasks, but there is still much room to improve on incorporating natural language guidance. The code and data are publicly available at https://github.com/tidalove/molmo2fish.