# ViewSAM: Learning View-aware Cross-modal Semantics for Weakly Supervised Cross-view Referring Multi-Object Tracking

arXiv: https://arxiv.org/abs/2605.02638 · HTML: https://mlsanity.com/paper/2605.02638/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 4 May 2026.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Omni-Scale Feature Learning for Person Re-Identification](https://mlsanity.com/paper/1905.00953/) (2019)
- [Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark](https://mlsanity.com/paper/2103.16746/) (2021)
- [SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory](https://mlsanity.com/paper/2411.11922/) (2024)
- [Referring Multi-Object Tracking](https://mlsanity.com/paper/2303.03366/) (2023)
- [Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping](https://mlsanity.com/paper/2305.11003/) (2023)
- [SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree](https://mlsanity.com/paper/2410.16268/) (2024)
- [Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark](https://mlsanity.com/paper/2306.02898/) (2023)
- [Joint Visual Grounding and Tracking with Natural Language Specification](https://mlsanity.com/paper/2303.12027/) (2023)
- [Divert More Attention to Vision-Language Tracking](https://mlsanity.com/paper/2207.01076/) (2022)
- [iKUN: Speak to Trackers without Retraining](https://mlsanity.com/paper/2312.16245/) (2023)
- [CC-3DT: Panoramic 3D Object Tracking via Cross-Camera Fusion](https://mlsanity.com/paper/2212.01247/) (2022)
- [TANGO: Training-free Embodied AI Agents for Open-world Tasks](https://mlsanity.com/paper/2412.10402/) (2024)
- [DIVOTrack: A Novel Dataset and Baseline Method for Cross-View Multi-Object Tracking in DIVerse Open Scenes](https://mlsanity.com/paper/2302.07676/) (2023)
- [SAM2MOT: A Novel Paradigm of Multi-Object Tracking by Segmentation](https://mlsanity.com/paper/2504.04519/) (2025)
- [Cross-View Referring Multi-Object Tracking](https://mlsanity.com/paper/2412.17807/) (2024)
- [WeakSAM: Segment Anything Meets Weakly-supervised Instance-level Recognition](https://mlsanity.com/paper/2402.14812/) (2024)
- [Beyond Visual Cues: Synchronously Exploring Target-Centric Semantics for Vision-Language Tracking](https://mlsanity.com/paper/2311.17085/) (2023)
- [LaMOT: Language-Guided Multi-Object Tracking](https://mlsanity.com/paper/2406.08324/) (2024)
- [R1-Track: Direct Application of MLLMs to Visual Object Tracking via Reinforcement Learning](https://mlsanity.com/paper/2506.21980/) (2025)
- [Temporal-Enhanced Multimodal Transformer for Referring Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/2410.13437/) (2024)
- [Cognitive Disentanglement for Referring Multi-Object Tracking](https://mlsanity.com/paper/2503.11496/) (2025)

## Built on by (most cited first, 2 of 2)

- [ReWorld-Track: A Recursive Event World Model for Language-Guided Multi-Camera Tracking](https://mlsanity.com/paper/2609.36677/) (2026)
- [EventLens: Event-Structure Reinforcement Learning for Video Understanding](https://mlsanity.com/paper/2610.16172/) (2026)

## Abstract (arXiv)

Cross-view Referring Multi-Object Tracking (CRMOT) aims to track multiple objects specified by natural language across multiple camera views, with globally consistent identities. Despite recent progress, existing methods rely heavily on costly frame-level spatial annotations and cross-view identity supervision. To reduce such reliance, we explore CRMOT under weak supervision by leveraging the capabilities of foundation models. However, our empirical study shows that directly applying foundation models such as SAM2 and SAM3, even with task-specific modifications, fails to accurately understand referring expressions and maintain consistent identities across views. Yet, they remain effective at producing reliable object tracklets that can serve as pseudo supervision. We therefore repurpose foundation models as pseudo-label generators and propose a two-stage framework for weakly supervised CRMOT, using only object category labels as coarse-grained supervision. In the first stage, we design an Affinity-guided Cross-view Re-prompting strategy to refine and associate SAM3-generated tracklets across cameras, producing reliable cross-view pseudo labels for subsequent training. In the second stage, we introduce ViewSAM, a CRMOT model built upon SAM2 that explicitly models view-aware cross-modal semantics. By formulating view-induced variations as learnable conditions, ViewSAM bridges the gap between view-variant visual observations and view-invariant textual expressions, enabling robust cross-view referring tracking with only approximately 10% additional parameters. Extensive experiments demonstrate that ViewSAM achieves SOTA performance under weak supervision and remains competitive with fully supervised methods.