mlsanityData to 14 Oct 2026

TrackVLA: Embodied Visual Tracking in the Wild

arXiv 2505.23189 ↗First posted 29 May 2025cs.RO cs.CVcited by 47

Not on the map. Why?

Abstract (arXiv)

Embodied visual tracking is a fundamental skill in Embodied AI, enabling an agent to follow a specific target in dynamic environments using only egocentric vision. This task is inherently challenging as it requires both accurate target recognition and effective trajectory planning under conditions of severe occlusion and high scene dynamics. Existing approaches typically address this challenge through a modular separation of recognition and planning. In this work, we propose TrackVLA, a Vision-Language-Action (VLA) model that learns the synergy between object recognition and trajectory planning. Leveraging a shared LLM backbone, we employ a language modeling head for recognition and an anchor-based diffusion model for trajectory planning. To train TrackVLA, we construct an Embodied Visual Tracking Benchmark (EVT-Bench) and collect diverse difficulty levels of recognition samples, resulting in a dataset of 1.7 million samples. Through extensive experiments in both synthetic and real-world environments, TrackVLA demonstrates SOTA performance and strong generalizability. It significantly outperforms existing methods on public benchmarks in a zero-shot manner while remaining robust to high dynamics and occlusion in real-world scenarios at 10 FPS inference speed. Our project page is: https://pku-epic.github.io/TrackVLA-web.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 61 of its 79 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

61 of 79 references.

References · 61

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2020Denoising Diffusion Implicit Modelscited by 9,781
  3. 2023Segment Anythingcited by 8,234
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  9. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  11. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  12. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
38 more
  1. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  2. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  3. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  4. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  5. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  6. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  7. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  8. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  9. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  10. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  11. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  12. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  13. 2019Learning Discriminative Model Prediction for Trackingcited by 344
  14. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 334
  15. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
  16. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 248
  17. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  18. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  19. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211
  20. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  21. 2022GNM: A General Navigation Model to Drive Any Robotcited by 171
  22. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 157
  23. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 156
  24. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  25. 2021Core Challenges of Social Robot Navigation: A Surveycited by 106
  26. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 98
  27. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 89
  28. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  29. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Modelscited by 67
  30. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 58
  31. 2023Principles and Guidelines for Evaluating Social Robot Navigation Algorithmscited by 56
  32. 2023PLIP: Language-Image Pre-training for Person Representation Learningcited by 49
  33. 20223D-Aware Object Goal Navigation via Simultaneous Exploration and Identificationcited by 46
  34. 2018End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learningcited by 35
  35. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 33
  36. 2021Towards Distraction-Robust Active Visual Trackingcited by 30
  37. 2022Offline Reinforcement Learning for Visual Navigationcited by 29
  38. 2023Follow Anything: Open-set detection, tracking, and following in real-timecited by 29

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 33.

47 papers.

Sort:

By area:

  1. 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Roboticscited by 127
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  3. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  4. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 44
  5. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 25
  6. 2025ActiveUMI: Robotic Manipulation with Active Perception from Robot-Free Human Demonstrationscited by 19
  7. 2025TrackVLA++: Unleashing Reasoning and Memory Capabilities in VLA Models for Embodied Visual Trackingcited by 17
  8. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  9. 2025SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigationcited by 12
  10. 2025LOVON: Legged Open-Vocabulary Object Navigatorcited by 11
  11. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 11
  12. 2025Point What You Mean: Visually Grounded Instruction Policycited by 10
35 more
  1. 2026VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memorycited by 10
  2. 2026TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environmentscited by 10
  3. 2026Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigationcited by 10
  4. 2025Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoningcited by 9
  5. 2025Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Compositioncited by 9
  6. 2026NavDreamer: Video Models as Zero-Shot 3D Navigatorscited by 9
  7. 2026ABot-N1: Toward a General Visual Language Navigation Foundation Modelcited by 6
  8. 2026CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Modelscited by 5
  9. 2026UAV-Track VLA: Embodied Aerial Tracking via Vision-Language-Action Modelscited by 4
  10. 2025MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learningcited by 3
  11. 2025UrbanVLA: A Vision-Language-Action Model for Urban Micromobilitycited by 3
  12. 2026Structured Observation Language for Efficient and Generalizable Vision-Language Navigationcited by 3
  13. 2026Instance-level Visual Active Tracking with Occlusion-Aware Planningcited by 3
  14. 2025DiffVL: Diffusion-Based Visual Localization on 2D Maps via BEV-Conditioned GPS Denoisingcited by 2
  15. 2026AsyncShield: A Plug-and-Play Edge Adapter for Asynchronous Cloud-based VLA Navigationcited by 2
  16. 2026Towards Backdoor-Based Ownership Verification for Vision-Language-Action Modelscited by 2
  17. 2026TMR-VLA:Vision-Language-Action Model for Magnetic Motion Control of Tri-leg Silicone-based Soft Robotcited by 1
  18. 2026A Vision-Language-Action Model for Adaptive Ultrasound-Guided Needle Insertion and Needle Trackingcited by 1
  19. 2026USS: Unifying Spatial-Semantic Prompting for End to End Embodied Visual Trackingcited by 1
  20. 2026CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Trackingcited by 1
  21. 2026LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigationcited by 1
  22. 2026Thinking with Cameras: Active Visual Reasoning via Dynamic Viewpoint Control for Surveillance Video Understandingcited by 1
  23. 2026TANGO: Humanoid Navigation in Cluttered Environments with a Whole-Body Vision-Language-Action Modelcited by 1
  24. 2026BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusioncited by 0
  25. 2026Mitigating State Aliasing in Vision-Language-Action Models via Inverse Dynamics Learningcited by 0
  26. 2026RescueBench: Can Embodied Agents Save Lives in the Wild ?cited by 0
  27. 2026EffiNav: Fusing Depth and Vision-Language for Efficient Object Goal Navigationcited by 0
  28. 2026Navigating the Proximity-Safety Balance: Constraint Decomposition for Human Following in Pedestrian Crowdscited by 0
  29. 2026MulDP: Multimodal Diffusion Policy for Autonomous Quadruped Parkour Navigation across Complex Terrainscited by 0
  30. 2026CST-WM: A Causally Structured World Model for Embodied Visual Trackingcited by 0
  31. 2026Visual Cue Guided Video Planning for Generalizable Robot Navigationcited by 0
  32. 2026UAVs Meet Embodied Intelligence: Bridging Human Intents and Flying Dynamics Via Harnessing Physical-Digital AI Agentscited by 0
  33. 2026RECAST: Recasting Vision-Language Semantics into an Actionable Cost Map for Robot Navigationcited by 0
  34. 2026UniTrackPLA: Unified Panorama-Language-Action Model for Instruction-Guided Navigation and Dynamic Person Trackingcited by 0
  35. 2026VLX-Go: Robust Embodied Tracking through Multi-Stage Learning and On-Policy Refinementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ReferTrack: Referring Then Tracking for Embodied Visual Tracking
  2. 2025VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models
  3. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
  4. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
  5. 2024A Survey on Vision-Language-Action Models for Embodied AI
  6. 2026EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models
  7. 2025Hierarchical Instruction-aware Embodied Visual Tracking
  8. 2026EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents