# From Technical Metrics to User Perception: A User Study of a Multimodal Human-Robot Interaction System for Object Detection and Grasping

arXiv: https://arxiv.org/abs/2607.00530 · HTML: https://mlsanity.com/paper/2607.00530/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [Physically Grounded Vision-Language Models for Robotic Manipulation](https://mlsanity.com/paper/2309.02561/) (2023)
- [Language and Sketching: An LLM-driven Interactive Multimodal Multitask Robot Navigation Framework](https://mlsanity.com/paper/2311.08244/) (2023)
- [Interactive Navigation in Environments with Traversable Obstacles Using Large Language and Vision-Language Models](https://mlsanity.com/paper/2310.08873/) (2023)
- [Leveraging Large Language Models in Human-Robot Interaction: A Critical Analysis of Potential and Pitfalls](https://mlsanity.com/paper/2405.00693/) (2024)
- [The human intention. A taxonomy attempt and its applications to robotics](https://mlsanity.com/paper/2602.15963/) (2026)
- [When the Inference Meets the Explicitness or Why Multimodality Can Make Us Forget About the Perfect Predictor](https://mlsanity.com/paper/2602.18850/) (2026)
- [An Approach to Combining Video and Speech with Large Language Models in Human-Robot Interaction](https://mlsanity.com/paper/2602.20219/) (2026)
- [A ROS 2 Wrapper for Florence-2: Multi-Mode Local Vision-Language Inference for Robotic Systems](https://mlsanity.com/paper/2604.01179/) (2026)

## Abstract (arXiv)

Improvements in the technical performance of human--robot interaction (HRI) systems do not automatically translate into differences that human users can detect during live interaction. This paper investigates whether a 15 percentage point gain in end-to-end task success (from 75% in a multimodal baseline system to 90% in an improved configuration identified through a prior ablation study) is sufficient to produce consistent and measurable differences in user perception. The baseline system combines Whisper for speech recognition, Florence-2 for open-vocabulary object detection, LLaMA 3.1 for action extraction, and an interval Type-2 fuzzy logic controller for motion execution. The improved configuration replaces the perception and language modules with Grounding DINO + SAM and Qwen 3.5 9B, respectively, while retaining the same controller. A within-subject user study with 24 participants compared both systems on the same tabletop object-grasping task. After interacting with each configuration, participants rated perceived speed, reliability, and overall competence and fluency on a 7-point Likert scale. Results show that 17 out of 24 participants (70.83%) preferred the improved system (exact binomial test, p = 0.043, h = 0.43), and all three perceptual constructs were rated significantly higher for the improved configuration after Holm correction, with large to very large effect sizes (p < 0.001). These findings confirm that the identified technical improvements are perceptible to users in direct interaction and underscore the importance of complementing benchmark evaluation with user-centred evidence when assessing robotic manipulation pipelines.