mlsanityData to 14 Oct 2026

Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks

arXiv 2401.14159 ↗First posted 25 Jan 2024cs.CVcited by 929

On the map: World models → World modeling from LiDAR generation

Abstract (arXiv)

We introduce Grounded SAM, which uses Grounding DINO as an open-set object detector to combine with the segment anything model (SAM). This integration enables the detection and segmentation of any regions based on arbitrary text inputs and opens a door to connecting various vision models. As shown in Fig.1, a wide range of vision tasks can be achieved by using the versatile Grounded SAM pipeline. For example, an automatic annotation pipeline based solely on input images can be realized by incorporating models such as BLIP and Recognize Anything. Additionally, incorporating Stable-Diffusion allows for controllable image editing, while the integration of OSX facilitates promptable 3D human motion analysis. Grounded SAM also shows superior performance on open-vocabulary benchmarks, achieving 48.7 mean AP on SegInW (Segmentation in the wild) zero-shot benchmark with the combination of Grounding DINO-Base and SAM-Huge models.

Key facts. Cited by 929 papers in the map (data through 14 Oct 2026). Placed in World modeling from LiDAR generation, one of 11 directions in World models. Builds on 69 of its 81 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

69 of 81 references.

Closest ancestors · 1

Same direction, newest first

  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks4 Jun

Other references · 68

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
38 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  3. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  6. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  7. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  8. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 823
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  10. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  12. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 573
  13. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  14. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  15. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 488
  16. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  17. 2022Blended Latent Diffusioncited by 432
  18. 2021Conditional DETR for Fast Training Convergencecited by 424
  19. 2023Segment Anything in High Qualitycited by 376
  20. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  21. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 352
  22. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  23. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 311
  24. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  25. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  26. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  27. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 227
  28. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  29. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  30. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  31. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192
  32. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 188
  33. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  34. 2021Mask2Former for Video Instance Segmentationcited by 171
  35. 2022DETRs with Hybrid Matchingcited by 150
  36. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  37. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 139
  38. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 130

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 188; 2025: 424; 2026: 317.

929 papers.

Sort:

By area:

  1. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  2. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
  3. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusioncited by 202
  4. 2025World Simulation with Video Foundation Models for Physical AIcited by 200
  5. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  6. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  7. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 158
  8. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 152
  9. 2024MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generationcited by 117
  10. 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 111
  11. 2025CAST: Component-Aligned 3D Scene Reconstruction from an RGB Imagecited by 106
  12. 2025Aether: Geometric-Aware Unified World Modelingcited by 97
38 more
  1. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  2. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 95
  3. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 89
  4. 2025PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videoscited by 87
  5. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 84
  6. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 82
  7. 2024EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learningcited by 75
  8. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 69
  9. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 67
  10. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 66
  11. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 66
  12. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 65
  13. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 65
  14. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  15. 2024LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoningcited by 63
  16. 2024DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videoscited by 63
  17. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 63
  18. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 61
  19. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 61
  20. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 54
  21. 2024GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplanecited by 53
  22. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
  23. 2024FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localizationcited by 51
  24. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 51
  25. 2024VCR-GauS: View Consistent Depth-Normal Regularizer for Gaussian Surface Reconstructioncited by 50
  26. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 49
  27. 2025DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understandingcited by 49
  28. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systemscited by 48
  29. 2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 47
  30. 2024GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understandingcited by 46
  31. 2024TAPTR: Tracking Any Point with Transformers as Detectioncited by 45
  32. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  33. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  34. 2024SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentationcited by 44
  35. 2024Segment Anything in Medical Images and Videos: Benchmark and Deploymentcited by 42
  36. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  37. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  38. 2024Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Searchcited by 40

50 most cited of 929 All 929 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection
  2. 20253EED: Ground Everything Everywhere in 3D
  3. 2025Vision and Language Reference Prompt into SAM for Few-shot Segmentation
  4. 2026TempoGround: State-Aware Streaming Visual Grounding with Vision-Language Models
  5. 2026AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models
  6. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding
  7. 2025X-SAM: From Segment Anything to Any Segmentation
  8. 2026Training-Free Open-Vocabulary Visual Grounding for Remote Sensing Images and Videos