mlsanityData to 7 Oct 2026

Semantic Scene Difference Detection in Daily Life Patroling by Mobile Robots using Pre-Trained Large-Scale Vision-Language Model

arXiv 2309.16552 ↗First posted 28 Sep 2023cs.ROcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

It is important for daily life support robots to detect changes in their environment and perform tasks. In the field of anomaly detection in computer vision, probabilistic and deep learning methods have been used to calculate the image distance. These methods calculate distances by focusing on image pixels. In contrast, this study aims to detect semantic changes in the daily life environment using the current development of large-scale vision-language models. Using its Visual Question Answering (VQA) model, we propose a method to detect semantic changes by applying multiple questions to a reference image and a current image and obtaining answers in the form of sentences. Unlike deep learning-based methods in anomaly detection, this method does not require any training or fine-tuning, is not affected by noise, and is sensitive to semantic state changes in the real world. In our experiments, we demonstrated the effectiveness of this method by applying it to a patrol task in a real-life environment using a mobile robot, Fetch Mobile Manipulator. In the future, it may be possible to add explanatory power to changes in the daily life environment through spoken language.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 30 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

21 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors10 Mar
  2. 2022Visual Language Maps for Robot Navigation11 Oct

Other references · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2022Generative Adversarial Networkscited by 5,603
  8. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  11. 2021Grounded Language-Image Pre-trainingcited by 1,147
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
7 more
  1. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  2. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  3. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  4. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  5. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  6. 2018Neural Modular Control for Embodied Question Answeringcited by 56
  7. 2023VQA-based Robotic State Recognition Optimized with Genetic Algorithmcited by 10

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024Foundation Models for Autonomous Robots in Unstructured Environmentscited by 3
  2. 2024Semantic Layering in Room Segmentation via LLMscited by 1
  3. 2024Zero-shot Degree of Ill-posedness Estimation for Active Small Object Change Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Semantic Anomaly Detection with Large Language Models
  2. 2023Binary State Recognition by Robots using Visual Question Answering of Pre-Trained Vision-Language Model
  3. 2026RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
  4. 2024Effectiveness Assessment of Recent Large Vision-Language Models
  5. 2025SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Models
  6. 2025Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets
  7. 2024Enhancing Robot Explanation Capabilities through Vision-Language Models: a Preliminary Study by Interpreting Visual Inputs for Improved Human-Robot Interaction
  8. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey