mlsanityData to 13 Oct 2026

VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation

arXiv 2403.12415 ↗First posted 19 Mar 2024cs.CV cs.HCcited by 13

Not on the map. Why?

Abstract (arXiv)

This paper explores the potential of Large Language Models(LLMs) in zero-shot anomaly detection for safe visual navigation. With the assistance of the state-of-the-art real-time open-world object detection model Yolo-World and specialized prompts, the proposed framework can identify anomalies within camera-captured frames that include any possible obstacles, then generate concise, audio-delivered descriptions emphasizing abnormalities, assist in safe visual navigation in complex circumstances. Moreover, our proposed framework leverages the advantages of LLMs and the open-vocabulary object detection model to achieve the dynamic scenario switch, which allows users to transition smoothly from scene to scene, which addresses the limitation of traditional visual navigation. Furthermore, this paper explored the performance contribution of different prompt components, provided the vision for future improvement in visual accessibility, and paved the way for LLMs in video anomaly detection and vision-language understanding.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Builds on 42 of its 71 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

42 of 71 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2023GPT-4 Technical Reportcited by 18,278
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  5. 2023Visual Instruction Tuningcited by 7,827
  6. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,840
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,299
  9. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,312
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,236
  11. 2021Grounded Language-Image Pre-trainingcited by 1,152
  12. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,063
30 more
  1. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  2. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 635
  3. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 615
  4. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  5. 2019Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selectioncited by 539
  6. 2021Image Segmentation Using Text and Image Promptscited by 511
  7. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 500
  8. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 437
  9. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  10. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  11. 2018Zero-Shot Object Detectioncited by 189
  12. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  13. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  14. 2019On Extractive and Abstractive Neural Document Summarization with Transformer Language Modelscited by 83
  15. 2019Zero-shot Text Classification With Generative Language Modelscited by 55
  16. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  17. 2023Better Zero-Shot Reasoning with Self-Adaptive Promptingcited by 52
  18. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  19. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  20. 2023Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Leadcited by 40
  21. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  22. 2023GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Modelscited by 32
  23. 2024VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Modelscited by 23
  24. 2023Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classificationcited by 18
  25. 2023Harnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectivescited by 16
  26. 2023Video OWL-ViT: Temporally-consistent open-world localization in videocited by 13
  27. 2024NavHint: Vision and Language Navigation Agent with a Hint Generatorcited by 12
  28. 2024Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossingcited by 12
  29. 2023LLM-based Smart Reply (LSR): Enhancing Collaborative Performance with ChatGPT-mediated Smart Reply Systemcited by 8
  30. 2023Interactive Navigation in Environments with Traversable Obstacles Using Large Language and Vision-Language Modelscited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024Large Language Models for Manufacturingcited by 24
  2. 2024WalkVLM:Aid Visually Impaired People Walking by Vision Language Modelcited by 13
  3. 2025Seeing and Reasoning with Confidence: Supercharging Multimodal LLMs with an Uncertainty-Aware Agentic Frameworkcited by 12
  4. 2025TANGO: Traversability-Aware Navigation with Local Metric Control for Topological Goalscited by 7
  5. 2024ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind Peoplecited by 5
  6. 2024HEADS-UP: Head-Mounted Egocentric Dataset for Trajectory Prediction in Blind Assistance Systemscited by 3
  7. 2025A Multi-Agent Framework Integrating Large Language Models and Generative AI for Accelerated Metamaterial Designcited by 3
  8. 2024INVARLLM: LLM-assisted Physical Invariant Extraction for Cyber-Physical Systems Anomaly Detectioncited by 2
  9. 2025Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Controlcited by 2
  10. 2025VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Modelscited by 2
  11. 2025ALARM: Automated MLLM-Based Anomaly Detection in Complex-EnviRonment Monitoring with Uncertainty Quantificationcited by 2
  12. 2025How Blind and Low-Vision Individuals Prefer Large Vision-Language Model-Generated Scene Descriptionscited by 0
1 more
  1. 2026Time-Aware Assistive Navigationcited by 0