mlsanityData to 14 Oct 2026

From Objects to Events: Unlocking Complex Visual Understanding in Object Detectors via LLM-guided Symbolic Reasoning

arXiv 2502.05843 ↗First posted 9 Feb 2025cs.CVcited by 3

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Current object detectors excel at entity localization and classification, yet exhibit inherent limitations in event recognition capabilities. This deficiency arises from their architecture's emphasis on discrete object identification rather than modeling the compositional reasoning, inter-object correlations, and contextual semantics essential for comprehensive event understanding. To address this challenge, we present a novel framework that expands the capability of standard object detectors beyond mere object recognition to complex event understanding through LLM-guided symbolic reasoning. Our key innovation lies in bridging the semantic gap between object detection and event understanding without requiring expensive task-specific training. The proposed plug-and-play framework interfaces with any open-vocabulary detector while extending their inherent capabilities across architectures. At its core, our approach combines (i) a symbolic regression mechanism exploring relationship patterns among detected entities and (ii) a LLM-guided strategically guiding the search toward meaningful expressions. These discovered symbolic rules transform low-level visual perception into interpretable event understanding, providing a transparent reasoning path from objects to events with strong transferability across domains.We compared our training-free framework against specialized event recognition systems across diverse application domains. Experiments demonstrate that our framework enhances multiple object detector architectures to recognize complex events such as illegal fishing activities (75% AUROC, +8.36% improvement), construction safety violations (+15.77%), and abnormal crowd behaviors (+23.16%). Code is available at hrefhttps://github.com/MAC-AutoML/SymbolicDethere.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 59 of its 73 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

59 of 73 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Question Aware Vision Transformer for Multimodal Reasoning8 Feb
  2. 2023DetGPT: Detect What You Need via Reasoning23 May

Other references · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2020End-to-End Object Detection with Transformerscited by 7,414
  10. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
38 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  2. 2015Fast R-CNNcited by 2,933
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  4. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  5. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  6. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  7. 2021Grounded Language-Image Pre-trainingcited by 1,156
  8. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  9. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  11. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 573
  12. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  13. 2023Segment Everything Everywhere All at Oncecited by 548
  14. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  15. 2021Conditional DETR for Fast Training Convergencecited by 424
  16. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 372
  17. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  18. 2022Recurrent Vision Transformers for Object Detection with Event Camerascited by 159
  19. 2019Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoningcited by 151
  20. 2022DETRs with Hybrid Matchingcited by 150
  21. 2021OW-DETR: Open-world Detection Transformercited by 129
  22. 2018Explainable and Explicit Visual Reasoning over Scene Graphscited by 103
  23. 2020Bridging Knowledge Graphs to Generate Scene Graphscited by 86
  24. 2020A Background-Agnostic Framework with Adversarial Training for Abnormal Event Detection in Videocited by 86
  25. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  26. 2023Aligning and Prompting Everything All at Once for Universal Visual Perceptioncited by 60
  27. 2022PROB: Probabilistic Objectness for Open World Object Detectioncited by 57
  28. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  29. 2023Diversity-Measurable Anomaly Detectioncited by 41
  30. 2023SODFormer: Streaming Object Detection with Transformer Using Events and Framescited by 39
  31. 2021Learning to Detect Every Thing in an Open Worldcited by 37
  32. 2022GPS: Genetic Prompt Search for Efficient Few-shot Learningcited by 37
  33. 2023Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectorscited by 37
  34. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answeringcited by 32
  35. 2022RU-Net: Regularized Unrolling Network for Scene Graph Generationcited by 28
  36. 2023Neural-Logic Human-Object Interaction Detectioncited by 25
  37. 2022Motion Robust High-Speed Light-Weighted Object Detection With Event Cameracited by 23
  38. 2023Dual Memory Aggregation Network for Event-Based Object Detection with Learnable Representationcited by 16

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Decoupling Template Bias in CLIP: Harnessing Empty Prompts for Enhanced Few-Shot Learningcited by 4
  2. 2025HyCoVAD: A Hybrid SSL-LLM Model for Complex Video Anomaly Detectioncited by 0
  3. 2026Absence is Presence: Understanding Visual Scene Negative Events Under Safety Cognitive Constraintcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adaptive Event Stream Slicing for Open-Vocabulary Event-Based Object Detection via Vision-Language Knowledge Distillation
  2. 2024Towards Neuro-Symbolic Video Understanding
  3. 2023Towards A Unified Neural Architecture for Visual Recognition and Reasoning
  4. 2024An Empirical Evaluation of Neural and Neuro-symbolic Approaches to Real-time Multimodal Complex Event Detection
  5. 2025Video Event Reasoning and Prediction by Fusing World Knowledge from LLMs with Vision Foundation Models
  6. 2025FLORA: Formal Language Model Enables Robust Training-free Zero-shot Object Referring Analysis
  7. 2025VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models
  8. 2025LLM-Guided Agentic Object Detection for Open-World Understanding