Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimization
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
In order for robots to autonomously navigate and operate in diverse environments, it is essential for them to recognize the state of their environment. On the other hand, the environmental state recognition has traditionally involved distinct methods tailored to each state to be recognized. In this study, we perform a unified environmental state recognition for robots through the spoken language with pre-trained large-scale vision-language models. We apply Visual Question Answering and Image-to-Text Retrieval, which are tasks of Vision-Language Models. We show that with our method, it is possible to recognize not only whether a room door is open/closed, but also whether a transparent door is open/closed and whether water is running in a sink, without training neural networks or manual programming. In addition, the recognition accuracy can be improved by selecting appropriate texts from the set of prepared texts based on black-box optimization. For each state recognition, only the text set and its weighting need to be changed, eliminating the need to prepare multiple different models and programs, and facilitating the management of source code and computer resource. We experimentally demonstrate the effectiveness of our method and apply it to the recognition behavior on a mobile robot, Fetch.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 24 references that are in the map. First posted on arXiv 26 Sep 2024.
Builds on
16 of 24 references.
Closest ancestors · 2
Same direction, newest first
- 2023Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors10 Mar
- 2022Visual Language Maps for Robot Navigation11 Oct
Other references · 14
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023GPT-4 Technical Reportcited by 18,132
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
- 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
- 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
- 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
- 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
- 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
- 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
- 2023VQA-based Robotic State Recognition Optimized with Genetic Algorithmcited by 10
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Binary State Recognition by Robots using Visual Question Answering of Pre-Trained Vision-Language Model
- 2024Robotic State Recognition with Image-to-Text Retrieval Task of Pre-Trained Vision-Language Model and Black-Box Optimization
- 2024CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environments
- 2026RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
- 2026Lightweight Visual Reasoning for Socially-Aware Robots
- 2025QueryAdapter: Rapid Adaptation of Vision-Language Models in Response to Natural Language Queries
- 2026A Modular Vision-Language-Action Robotics Framework for Indoor Environments
- 2024EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answering