mlsanityData to 14 Oct 2026

Follow Anything: Open-set detection, tracking, and following in real-time

arXiv 2308.05737 ↗First posted 10 Aug 2023cs.RO cs.CV cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Tracking and following objects of interest is critical to several robotics use cases, ranging from industrial automation to logistics and warehousing, to healthcare and security. In this paper, we present a robotic system to detect, track, and follow any object in real-time. Our approach, dubbed ``follow anything'' (FAn), is an open-vocabulary and multimodal model -- it is not restricted to concepts seen at training time and can be applied to novel classes at inference time using text, images, or click queries. Leveraging rich visual descriptors from large-scale pre-trained models (foundation models), FAn can detect and segment objects by matching multimodal queries (text, images, clicks) against an input image sequence. These detected and segmented objects are tracked across image frames, all while accounting for occlusion and object re-emergence. We demonstrate FAn on a real-world robotic system (a micro aerial vehicle) and report its ability to seamlessly follow the objects of interest in a real-time control loop. FAn can be deployed on a laptop with a lightweight (6-8 GB) graphics card, achieving a throughput of 6-20 frames per second. To enable rapid adoption, deployment, and extensibility, we open-source all our code on our project webpage at https://github.com/alaamaalouf/FollowAnything . We also encourage the reader to watch our 5-minutes explainer video in this https://www.youtube.com/watch?v=6Mgt3EPytrw .

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 47 of its 67 references that are in the map. First posted on arXiv 10 Aug 2023.

Builds on

47 of 67 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2023Segment Anythingcited by 8,234
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  10. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  11. 2015Fast R-CNNcited by 2,933
  12. 2018YOLOv3: An Incremental Improvementcited by 2,659
35 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  3. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,630
  6. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,020
  7. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  8. 2022Language-driven Semantic Segmentationcited by 717
  9. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  10. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  11. 2016R-FCN: Object Detection via Region-based Fully Convolutional Networkscited by 414
  12. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  13. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  14. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  15. 2023Fast Segment Anythingcited by 290
  16. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  17. 2018Fast Online Object Tracking and Segmentation: A Unifying Approachcited by 272
  18. 2020Experience Grounds Languagecited by 242
  19. 2021Associating Objects with Transformers for Video Object Segmentationcited by 226
  20. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  21. 2023Segment and Track Anythingcited by 193
  22. 2022Decoupling Features in Hierarchical Propagation for Video Object Segmentationcited by 148
  23. 2020See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networkscited by 138
  24. 2021Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusioncited by 129
  25. 2020Video Object Segmentation with Episodic Graph Memory Networkscited by 118
  26. 2020CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80cited by 106
  27. 2020Zero-Shot Video Object Segmentation via Attentive Graph Neural Networkscited by 84
  28. 2017Open Vocabulary Scene Parsingcited by 76
  29. 2019AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Ratescited by 49
  30. 2022Vision-based Anti-UAV Detection and Trackingcited by 44
  31. 2022Treating Motion as Option to Reduce Motion Dependency in Unsupervised Video Object Segmentationcited by 35
  32. 2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
  33. 2021A Unified Approach to Coreset Learningcited by 18
  34. 2022Deep Learning on Home Drone: Searching for the Optimal Architecturecited by 5
  35. 2022Vision-based system for a real-time detection and following of UAVcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 8; 2026: 9.

29 papers.

Sort:

By area:

  1. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  2. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 47
  3. 2024PTQ4SAM: Post-Training Quantization for Segment Anythingcited by 32
  4. 2024RoboHop: Segment-based Topological Map Representation for Open-World Visual Navigationcited by 25
  5. 2024Probing Multimodal LLMs as World Models for Drivingcited by 20
  6. 2024Revisit Anything: Visual Place Recognition via Image Segment Retrievalcited by 19
  7. 2025TrackVLA++: Unleashing Reasoning and Memory Capabilities in VLA Models for Embodied Visual Trackingcited by 17
  8. 2024Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networkscited by 14
  9. 2024Open-Set Recognition in the Age of Vision-Language Modelscited by 13
  10. 2024Open-World Drone Active Tracking with Goal-Centered Rewardscited by 6
  11. 2025SINGER: An Onboard Generalist Vision-Language Navigation Policy for Dronescited by 6
  12. 2026CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Modelscited by 5
17 more
  1. 2026See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selectioncited by 4
  2. 2024Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Featurescited by 3
  3. 2025AHCQ-SAM: Toward Accurate and Hardware-Compatible Post-Training Segment Anything Model Quantizationcited by 3
  4. 2025Decentralized Vision-Based Autonomous Aerial Wildlife Monitoringcited by 3
  5. 2026Instance-level Visual Active Tracking with Occlusion-Aware Planningcited by 3
  6. 2024Watching Swarm Dynamics from Above: A Framework for Advanced Object Tracking in Drone Videoscited by 2
  7. 2026Robustness Is a Function, Not a Number: A Factorized Comprehensive Study of OOD Robustness in Vision-Based Drivingcited by 2
  8. 2025NOVA: Navigation via Object-Centric Visual Autonomy for High-Speed Target Tracking in Unstructured GPS-Denied Environmentscited by 1
  9. 2025Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samplescited by 1
  10. 2024Track Anything Rapter(TAR)cited by 0
  11. 2024MSEG-VCUQ: Multimodal SEGmentation with Enhanced Vision Foundation Models, Convolutional Neural Networks, and Uncertainty Quantification for High-Speed Video Phase Detection Datacited by 0
  12. 2025HQ-SMem: Video Segmentation and Tracking Using Memory Efficient Object Embedding With Selective Update and Self-Supervised Distillation Feedbackcited by 0
  13. 2026Do Open-Vocabulary Detectors Transfer to Aerial Imagery? A Comparative Evaluationcited by 0
  14. 2026Banshee: Target Switch Attacks on Gimbal-Stabilized Visual Tracking Systems via Acoustic Injectioncited by 0
  15. 2026Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fieldscited by 0
  16. 2026CST-WM: A Causally Structured World Model for Embodied Visual Trackingcited by 0
  17. 2026Mind the Temporal Gap: Evaluating Video Models for Temporal Understanding in Drivingcited by 0