mlsanityData to 14 Oct 2026

HuBo-VLM: Unified Vision-Language Model designed for HUman roBOt interaction tasks

arXiv 2308.12537 ↗First posted 24 Aug 2023cs.RO cs.CVcited by 6

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Human robot interaction is an exciting task, which aimed to guide robots following instructions from human. Since huge gap lies between human natural language and machine codes, end to end human robot interaction models is fair challenging. Further, visual information receiving from sensors of robot is also a hard language for robot to perceive. In this work, HuBo-VLM is proposed to tackle perception tasks associated with human robot interaction including object detection and visual grounding by a unified transformer based vision language model. Extensive experiments on the Talk2Car benchmark demonstrate the effectiveness of our approach. Code would be publicly available in https://github.com/dzcgaara/HuBo-VLM.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 41 references that are in the map. First posted on arXiv 24 Aug 2023.

Builds on

32 of 41 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
20 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2018Deep contextualized word representationscited by 2,290
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  6. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  10. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  11. 2016Modeling Context in Referring Expressionscited by 1,063
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  14. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  15. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  16. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  17. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  18. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  19. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 142
  20. 2020What Makes for End-to-End Object Detection?cited by 49

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 2.

6 papers.

  1. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 43
  2. 2023GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Modelscited by 32
  3. 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigationcited by 10
  4. 2024Empowering Visual Creativity: A Vision-Language Assistant to Image Editing Recommendationscited by 6
  5. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactionscited by 6
  6. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Model
  2. 2025Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Models
  3. 2024A3VLM: Actionable Articulation-Aware Vision Language Model
  4. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
  5. 2026The World According to a Social Robot -- Augmenting Human-Robot Dialogue With Vision Language Models
  6. 2026Monocular 3D Object Position Estimation with VLMs for Human-Robot Interaction
  7. 2024TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models
  8. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation