mlsanityData to 14 Oct 2026

Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos

arXiv 2506.05302 ↗First posted 5 Jun 2025cs.CVcited by 51

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Perceive Anything Model (PAM), a conceptually straightforward and efficient framework for comprehensive region-level visual understanding in images and videos. Our approach extends the powerful segmentation model SAM 2 by integrating Large Language Models (LLMs), enabling simultaneous object segmentation with the generation of diverse, region-specific semantic outputs, including categories, label definition, functional explanations, and detailed captions. A key component, Semantic Perceiver, is introduced to efficiently transform SAM 2's rich visual features, which inherently carry general vision, localization, and semantic priors into multi-modal tokens for LLM comprehension. To support robust multi-granularity understanding, we also develop a dedicated data refinement and augmentation pipeline, yielding a high-quality dataset of 1.5M image and 0.6M video region-semantic annotations, including novel region-level streaming video caption data. PAM is designed for lightweightness and efficiency, while also demonstrates strong performance across a diverse range of region understanding tasks. It runs 1.2-2.4x faster and consumes less GPU memory than prior approaches, offering a practical solution for real-world applications. We believe that our effective approach will serve as a strong baseline for future research in region-level visual understanding.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 59 of its 79 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

59 of 79 references.

References · 59

  1. 2023Segment Anythingcited by 8,234
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
38 more
  1. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  2. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  5. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  6. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 383
  7. 2023Segment Anything in High Qualitycited by 376
  8. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 352
  9. 2019Abductive Commonsense Reasoningcited by 301
  10. 2023Fast Segment Anythingcited by 290
  11. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  12. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  13. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 236
  14. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  15. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  16. 2023Segment and Track Anythingcited by 193
  17. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 171
  18. 2017Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognitioncited by 162
  19. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  20. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  21. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  22. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  23. 2019ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVTcited by 122
  24. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 121
  25. 2020Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentencescited by 118
  26. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  27. 2018Actor and Action Video Segmentation from a Sentencecited by 106
  28. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  29. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 99
  30. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  31. 2024SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memorycited by 94
  32. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 89
  33. 2020Human-centric Spatio-Temporal Video Grounding With Visual Transformerscited by 85
  34. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  35. 2024Streaming Dense Video Captioningcited by 82
  36. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 72
  37. 2016Online Action Detectioncited by 69
  38. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 29; 2026: 22.

51 papers.

Sort:

By area:

  1. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 67
  2. 2025Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learningcited by 34
  3. 2025VABench: A Comprehensive Benchmark for Audio-Video Generationcited by 30
  4. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  5. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 25
  6. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  7. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 20
  8. 2025SCALAR: Scale-wise Controllable Visual Autoregressive Learningcited by 18
  9. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  10. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  11. 2025VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMscited by 13
  12. 2026Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D Worldcited by 12
38 more
  1. 2025Text2SQL-Flow: A Robust SQL-Aware Data Augmentation Framework for Text-to-SQLcited by 11
  2. 2026SAMTok: Representing Any Mask with Two Wordscited by 11
  3. 2025MICo-150K: A Comprehensive Dataset Advancing Multi-Image Compositioncited by 10
  4. 2025Scone: Bridging Composition and Distinction in Subject-Driven Image Generation via Unified Understanding-Generation Modelingcited by 10
  5. 2025From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoningcited by 9
  6. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  7. 2025Faithful-First Reasoning, Planning, and Acting for Multimodal LLMscited by 7
  8. 2025Semantic Context Matters: Improving Conditioning for Autoregressive Modelscited by 7
  9. 2025Eevee: Towards Close-up High-resolution Video-based Virtual Try-oncited by 7
  10. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Datacited by 6
  11. 2025DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLMcited by 6
  12. 2026Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspectivecited by 6
  13. 2026CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generationcited by 6
  14. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  15. 2025Describe Anything Model for Visual Question Answering on Text-rich Imagescited by 5
  16. 2025DescribeEarth: Describe Anything for Remote Sensing Imagescited by 5
  17. 2026GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environmentscited by 5
  18. 2026AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAGcited by 5
  19. 2025Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrievalcited by 4
  20. 2025Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Samplingcited by 4
  21. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  22. 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Taskscited by 3
  23. 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
  24. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  25. 2025Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Explorationcited by 2
  26. 2025VCU-Bridge: Hierarchical Visual Connotation Understanding via Semantic Bridgingcited by 2
  27. 2025UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Modelscited by 2
  28. 2025GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Modelscited by 2
  29. 2026AD-MIR: Bridging the Gap from Perception to Persuasion in Advertising Video Understanding via Structured Reasoningcited by 2
  30. 2026Chain of Mindset: Reasoning with Adaptive Cognitive Modescited by 2
  31. 2026TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understandingcited by 2
  32. 2026TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videoscited by 1
  33. 2026MotionAtlas: Detailed Region Captioning for Motion-Centric Videoscited by 1
  34. 2025RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehensioncited by 0
  35. 2026WOW-Seg: A Word-free Open World Segmentation Modelcited by 0
  36. 2026FRFDet: Efficient UAV Small Object Detection with Symmetric Sampling and Scalable Fusioncited by 0
  37. 2026RegRet: Enhancing Region-Level Retrieval in Large Multimodal Modelscited by 0
  38. 2026VIEWCRAFT: Evolving What Multimodal Agents See and How They Read Itcited by 0

50 most cited of 51 All 51 (JSON)