# YesTrack: Referring Multi-Object Tracking via MLLM-based Yes/No Verification

arXiv: https://arxiv.org/abs/2609.02318 · HTML: https://mlsanity.com/paper/2609.02318/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 2 Sep 2026.

## Builds on (most cited first, 30 of 32)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [Deep Learning for Person Re-identification: A Survey and Outlook](https://mlsanity.com/paper/2001.04193/) (2020)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://mlsanity.com/paper/1903.07071/) (2019)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking](https://mlsanity.com/paper/2203.14360/) (2022)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph](https://mlsanity.com/paper/2307.07697/) (2023)
- [FastReID: A Pytorch Toolbox for General Instance Re-identification](https://mlsanity.com/paper/2006.02631/) (2020)
- [Video Object Segmentation with Language Referring Expressions](https://mlsanity.com/paper/1803.08006/) (2018)
- [Talk2Car: Taking Control of Your Self-Driving Car](https://mlsanity.com/paper/1909.10838/) (2019)
- [RGB-Infrared Cross-Modality Person Re-Identification via Joint Pixel and Feature Alignment](https://mlsanity.com/paper/1910.05839/) (2019)
- [Referring Multi-Object Tracking](https://mlsanity.com/paper/2303.03366/) (2023)
- [RF-DETR: Neural Architecture Search for Real-Time Detection Transformers](https://mlsanity.com/paper/2511.09554/) (2025)
- [iKUN: Speak to Trackers without Retraining](https://mlsanity.com/paper/2312.16245/) (2023)
- [Bootstrapping Referring Multi-Object Tracking](https://mlsanity.com/paper/2406.05039/) (2024)
- [EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2402.18302/) (2024)
- [When Large Vision-Language Models Meet Person Re-Identification](https://mlsanity.com/paper/2411.18111/) (2024)
- [ReferGPT: Towards Zero-Shot Referring Multi-Object Tracking](https://mlsanity.com/paper/2504.09195/) (2025)
- [LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re-Identification](https://mlsanity.com/paper/2504.10174/) (2025)
- [MLS-Track: Multilevel Semantic Interaction in RMOT](https://mlsanity.com/paper/2404.12031/) (2024)
- [Temporal-Enhanced Multimodal Transformer for Referring Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/2410.13437/) (2024)
- [Cognitive Disentanglement for Referring Multi-Object Tracking](https://mlsanity.com/paper/2503.11496/) (2025)

## Abstract (arXiv)

Referring multi-object tracking (RMOT) aims to track every instance in a video that matches a given language expression. Despite the recent integration of multimodal large language models (MLLMs) to enhance generalization, existing methods predominantly relegate them to the role of caption generators, necessitating external modules for final decision-making. This paradigm not only introduces extra latency but also severely underutilizes the inherent vision-language alignment capabilities of MLLMs. To address these limitations, we propose YesTrack, a novel two-stage RMOT method that reformulates referring as a discriminative task, directly leveraging MLLMs for Yes/No verification without explicit text generation. To further enhance the reliability and efficiency of this MLLM-based verification, we introduce two lightweight temporal consistency constraints: Temporal Confidence Prior (TCP) and Temporal Reference Propagation (TRP). We further validate the generality of this discriminative paradigm by proposing YesTrack-MOT, a straightforward yet highly effective instantiation for generic multi-object tracking (MOT). Experiments on Refer-KITTI and Refer-KITTI-V2 show that YesTrack significantly outperforms existing state-of-the-art methods while maintaining high efficiency, even when implemented with the smallest variant of Qwen3-VL. Code is released at https://github.com/ggbondrighthere24/YesTrack.