# Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors

arXiv: https://arxiv.org/abs/2303.05674 · HTML: https://mlsanity.com/paper/2303.05674/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 10 Mar 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder](https://mlsanity.com/paper/1711.00614/) (2017)
- [Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions](https://mlsanity.com/paper/1710.06280/) (2017)
- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)
- [VQA-based Robotic State Recognition Optimized with Genetic Algorithm](https://mlsanity.com/paper/2303.05052/) (2023)
- [World Robot Challenge 2020 -- Partner Robot: A Data-Driven Approach for Room Tidying with Mobile Manipulator](https://mlsanity.com/paper/2207.10106/) (2022)

## Built on by (most cited first, 12 of 12)

- [Real-World Robot Applications of Foundation Models: A Review](https://mlsanity.com/paper/2402.05741/) (2024)
- [Semantic Scene Difference Detection in Daily Life Patroling by Mobile Robots using Pre-Trained Large-Scale Vision-Language Model](https://mlsanity.com/paper/2309.16552/) (2023)
- [Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimization](https://mlsanity.com/paper/2403.08239/) (2024)
- [Foundation Models for Autonomous Robots in Unstructured Environments](https://mlsanity.com/paper/2407.14296/) (2024)
- [CLIP-Clique: Graph-based Correspondence Matching Augmented by Vision Language Models for Object-based Global Localization](https://mlsanity.com/paper/2410.03054/) (2024)
- [AnoleVLA: Lightweight Vision-Language-Action Model with Deep State Space Models for Mobile Manipulation](https://mlsanity.com/paper/2603.15046/) (2026)
- [Daily Assistive View Control Learning of Low-Cost Low-Rigidity Robot via Large-Scale Vision-Language Model](https://mlsanity.com/paper/2312.07451/) (2023)
- [VEglue: Testing Visual Entailment Systems via Object-Aligned Joint Erasing](https://mlsanity.com/paper/2403.02581/) (2024)
- [Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimization](https://mlsanity.com/paper/2409.17519/) (2024)
- [Robotic State Recognition with Image-to-Text Retrieval Task of Pre-Trained Vision-Language Model and Black-Box Optimization](https://mlsanity.com/paper/2410.22707/) (2024)
- [Monocular 3D Object Position Estimation with VLMs for Human-Robot Interaction](https://mlsanity.com/paper/2603.01224/) (2026)
- [The World According to a Social Robot -- Augmenting Human-Robot Dialogue With Vision Language Models](https://mlsanity.com/paper/2607.16318/) (2026)

## Abstract (arXiv)

In recent years, a number of models that learn the relations between vision and language from large datasets have been released. These models perform a variety of tasks, such as answering questions about images, retrieving sentences that best correspond to images, and finding regions in images that correspond to phrases. Although there are some examples, the connection between these pre-trained vision-language models and robotics is still weak. If they are directly connected to robot motions, they lose their versatility due to the embodiment of the robot and the difficulty of data collection, and become inapplicable to a wide range of bodies and situations. Therefore, in this study, we categorize and summarize the methods to utilize the pre-trained vision-language models flexibly and easily in a way that the robot can understand, without directly connecting them to robot motions. We discuss how to use these models for robot motion selection and motion planning without re-training the models. We consider five types of methods to extract information understandable for robots, and show the results of state recognition, object recognition, affordance recognition, relation recognition, and anomaly detection based on the combination of these five methods. We expect that this study will add flexibility and ease-of-use, as well as new applications, to the recognition behavior of existing robots.