mlsanityData to 13 Oct 2026

From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Grounded Open-vocabulary Situation Recognition

arXiv 2507.14686 ↗First posted 19 Jul 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent Multimodal Large Language Models (MLLMs) exhibit strong zero-shot abilities but struggle with complex Grounded Situation Recognition (GSR) and are resource-intensive for edge device deployment. Meanwhile, conventional GSR models often lack generalization ability, falling short in recognizing unseen and rare situations. In this paper, we exploit transferring knowledge from a teacher MLLM to a small GSR model to enhance its generalization and zero-shot abilities, thereby introducing the task of Open-vocabulary Grounded Situation Recognition (Ov-GSR). To achieve this, we propose Multimodal Interactive Prompt Distillation (MIPD), a novel framework that distills enriched multimodal knowledge from the foundation model, enabling the student Ov-GSR model to recognize unseen situations and be better aware of rare situations. Specifically, the MIPD framework first leverages the LLM-based Judgmental Rationales Generator (JRG) to construct positive and negative glimpse and gaze rationales enriched with contextual semantic information. The proposed scene-aware and instance-perception prompts are then introduced to align rationales with visual information from the MLLM teacher via the Negative-Guided Multimodal Prompting Alignment (NMPA) module, effectively capturing holistic and perceptual multimodal knowledge. Finally, the aligned multimodal knowledge is distilled into the student Ov-GSR model, providing a stronger foundation for generalization that enhances situation understanding, bridges the gap between seen and unseen scenarios, and mitigates prediction bias in rare cases. We evaluate MIPD on the refined Ov-SWiG dataset, achieving superior performance on seen, rare, and unseen situations, and further demonstrate improved unseen detection on the HICO-DET dataset.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 71 references that are in the map. First posted on arXiv 19 Jul 2025.

Builds on

42 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023DIME-FM: DIstilling Multimodal and Efficient Foundation Models31 Mar

Other references · 41

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  3. 2023Segment Anythingcited by 8,195
  4. 2020End-to-End Object Detection with Transformerscited by 7,396
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  6. 2024A Survey on LLM-as-a-Judgecited by 1,399
  7. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  8. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 746
  9. 2024Yi: Open Foundation Models by 01.AIcited by 715
  10. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  11. 2017Learning to Detect Human-Object Interactionscited by 238
  12. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
29 more
  1. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 127
  2. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
  3. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  4. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 87
  5. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  6. 2020Grounded Situation Recognitioncited by 78
  7. 2025Dataset Distillation with Neural Characteristic Function: A Minmax Perspectivecited by 45
  8. 2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
  9. 2022Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfercited by 33
  10. 2023The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understandingcited by 29
  11. 2023FreeKD: Knowledge Distillation via Semantic Frequency Promptcited by 27
  12. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  13. 2024VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Modelscited by 23
  14. 2022GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinementcited by 20
  15. 2021Rethinking the Two-Stage Framework for Grounded Situation Recognitioncited by 16
  16. 2021Grounded Situation Recognition with Transformerscited by 14
  17. 2022Efficient Image Captioning for Edge Devicescited by 14
  18. 2023Bitstream-Corrupted Video Recovery: A Novel Benchmark Dataset and Methodcited by 14
  19. 2024PromptMM: Multi-Modal Knowledge Distillation for Recommendation with Prompt-Tuningcited by 14
  20. 2024Empowering Large Language Model for Continual Video Question Answering with Collaborative Promptingcited by 14
  21. 2024Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Modelcited by 13
  22. 2023Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generationcited by 10
  23. 2024Seeing the Unseen: Visual Common Sense for Semantic Placementcited by 9
  24. 2024Seeing Beyond Classes: Zero-Shot Grounded Situation Recognition via Language Explainercited by 7
  25. 2023ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognitioncited by 6
  26. 2024CM2-Net: Continual Cross-Modal Mapping Network for Driver Action Recognitioncited by 6
  27. 2024OVMR: Open-Vocabulary Recognition with Multi-Modal Referencescited by 3
  28. 2025EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimizationcited by 3
  29. 2025Dynamic Scene Understanding from Vision-Language Representationscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  2. 2025Towards Understanding Visual Grounding in Visual Language Models
  3. 2026A Training-Free Guess What Vision Language Model from Snippets to Open-Vocabulary Object Detection
  4. 2024Visual Prompting in Multimodal Large Language Models: A Survey
  5. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation
  6. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervision
  7. 2024Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM
  8. 2026Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Models