mlsanityData to 14 Oct 2026

RELO: Reinforcement Learning to Localize for Visual Object Tracking

arXiv 2605.07379 ↗First posted 8 May 2026cs.CV cs.AIcited by 3

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Conventional visual object trackers localize targets using handcrafted spatial priors, often in the form of heatmaps. Such priors provide only surrogate supervision and are poorly aligned with tracking optimization and evaluation metrics, such as intersection over union (IoU) and area under the success curve (AUC). Here, we introduce RELO, a REinforcement-learning-to-LOcalize method for visual object tracking that formulates target localization as a Markov decision process. Specifically, RELO replaces handcrafted spatial priors with a localization policy learned over spatial positions via reinforcement learning, with rewards combining frame-level IoU and sequence-level AUC. We additionally introduce layer-aligned temporal token propagation to improve semantic consistency across frames, with negligible computational overhead. Across multiple benchmarks, RELO achieves superior results, attaining 57.5% AUC on LaSOText without template updates. This confirms that reward-driven localization provides an effective alternative to prior-driven localization for visual object tracking.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 48 of its 61 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

48 of 61 references.

References · 48

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
  7. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 475
  8. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 429
  9. 2021Transformer Trackingcited by 412
  10. 2021Learning Spatio-Temporal Transformer for Visual Trackingcited by 350
  11. 2019Learning Discriminative Model Prediction for Trackingcited by 344
  12. 2018TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wildcited by 343
36 more
  1. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 330
  2. 2022MixFormer: End-to-End Tracking with Iterative Mixed Attentioncited by 281
  3. 2020VarifocalNet: An IoU-aware Dense Object Detectorcited by 231
  4. 2021Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Trackingcited by 220
  5. 2019SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelinescited by 177
  6. 2017Need for Speed: A Benchmark for Higher Frame Rate Object Trackingcited by 137
  7. 2022AiATrack: Attention in Attention for Transformer Visual Trackingcited by 137
  8. 2020LaSOT: A High-quality Large-scale Single Object Tracking Benchmarkcited by 132
  9. 2024ODTrack: Online Dense Temporal Token Learning for Visual Trackingcited by 129
  10. 2021Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmarkcited by 126
  11. 2021SwinTrack: A Simple and Strong Baseline for Transformer Trackingcited by 126
  12. 2022Backbone is All Your Need: A Simplified Architecture for Visual Object Trackingcited by 119
  13. 2022Transforming Model Prediction for Trackingcited by 117
  14. 2022Transformer Tracking with Cyclic Shifting Window Attentioncited by 75
  15. 2024Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performancecited by 70
  16. 2024A Distractor-Aware Memory for Visual Object Tracking with SAM2cited by 66
  17. 2024OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuningcited by 65
  18. 2022Correlation-Aware Deep Trackingcited by 62
  19. 2023ARTrackV2: Prompting Autoregressive Tracker Where to Look and How to Describecited by 59
  20. 2023MixFormerV2: Efficient Fully Transformer Trackingcited by 57
  21. 2023Exploring Lightweight Hierarchical Vision Transformers for Efficient Visual Trackingcited by 53
  22. 2024SUTrack: Towards Simple and Unified Single Object Trackingcited by 52
  23. 2021Efficient Visual Tracking with Exemplar Transformerscited by 48
  24. 2021FEAR: Fast, Efficient, Accurate and Robust Visual Trackercited by 38
  25. 2017End-to-end Active Object Tracking via Reinforcement Learningcited by 28
  26. 2024VastTrack: Vast Category Visual Object Trackingcited by 26
  27. 2020Fast Template Matching and Update for Video Object Tracking and Segmentationcited by 24
  28. 2022Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migrationcited by 24
  29. 2025SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Trackingcited by 24
  30. 2025Exploring Decoupled Spatio-Temporal Consistency Learning and Self-Prompting Evolution for Self-Supervised Trackingcited by 23
  31. 2017Learning Policies for Adaptive Tracking with Deep Feature Cascadescited by 19
  32. 2025Two-stream Beats One-stream: Asymmetric Siamese Network for Efficient Visual Trackingcited by 15
  33. 2022Towards Sequence-Level Training for Visual Trackingcited by 13
  34. 2022Ranking-Based Siamese Visual Trackingcited by 10
  35. 2021Weakly-Supervised Domain Adaptation of Deep Regression Trackers via Reinforced Knowledge Distillationcited by 9
  36. 2018Correlation Filter Selection for Visual Tracking Using Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Models as Tools: An Agentic Coordination Framework for Unified Multimodal Visual Trackingcited by 0
  2. 2026Learning to Track from Privileged Target Appearancescited by 0
  3. 2026UniMergeTrack: Hierarchical Model Merging for General RGB-X Visual Object Trackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization
  2. 2025R1-Track: Direct Application of MLLMs to Visual Object Tracking via Reinforcement Learning
  3. 2025SAM2RL: Towards Reinforcement Learning Memory Control in Segment Anything Model 2
  4. 2026SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking
  5. 2026Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification
  6. 2026Motion-Aware Reinforcement Learning For Object Localization
  7. 2026ReMoT: Reinforcement Learning with Motion Contrast Triplets
  8. 2026SMTrack: State-Aware Mamba for Efficient Temporal Modeling in Visual Tracking