# Ablation Study of Multimodal Perception, Language Grounding, and Control for Human-Robot Interaction in an Object Detection and Grasping Task

arXiv: https://arxiv.org/abs/2605.00963 · HTML: https://mlsanity.com/paper/2605.00963/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 1 May 2026.

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [Physically Grounded Vision-Language Models for Robotic Manipulation](https://mlsanity.com/paper/2309.02561/) (2023)
- [Language and Sketching: An LLM-driven Interactive Multimodal Multitask Robot Navigation Framework](https://mlsanity.com/paper/2311.08244/) (2023)
- [Interactive Navigation in Environments with Traversable Obstacles Using Large Language and Vision-Language Models](https://mlsanity.com/paper/2310.08873/) (2023)
- [Leveraging Large Language Models in Human-Robot Interaction: A Critical Analysis of Potential and Pitfalls](https://mlsanity.com/paper/2405.00693/) (2024)
- [The human intention. A taxonomy attempt and its applications to robotics](https://mlsanity.com/paper/2602.15963/) (2026)
- [When the Inference Meets the Explicitness or Why Multimodality Can Make Us Forget About the Perfect Predictor](https://mlsanity.com/paper/2602.18850/) (2026)
- [An Approach to Combining Video and Speech with Large Language Models in Human-Robot Interaction](https://mlsanity.com/paper/2602.20219/) (2026)

## Abstract (arXiv)

This manuscript extends our previous multimodal human-robot interaction system by introducing a controlled ablation study of the three modules that most strongly influence end-to-end performance: the large language model used for action extraction, the perception system used for visual grounding, and the controller used for motion execution. The goal is not to redesign the full pipeline, but to isolate the contribution of each component under a common experimental protocol and then evaluate the best combinations end-to-end. We therefore compare three language models, five perception configurations, and three controllers, followed by a second-stage factorial study over the best candidates. The resulting analysis is intended to clarify which choices primarily affect execution time, which primarily affect success rate, and where the largest engineering gains are likely to come from in future revisions of the system.