# Human-in-the-loop Robotic Grasping using BERT Scene Representation

arXiv: https://arxiv.org/abs/2209.14026 · HTML: https://mlsanity.com/paper/2209.14026/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2022.

## Built on by (most cited first, 5 of 5)

- [Reasoning Grasping via Multimodal Large Language Model](https://mlsanity.com/paper/2402.06798/) (2024)
- [Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Grounding](https://mlsanity.com/paper/2301.11564/) (2023)
- [City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning](https://mlsanity.com/paper/2507.12795/) (2025)
- [Multi-Task Domain Adaptation for Language Grounding with 3D Objects](https://mlsanity.com/paper/2407.02846/) (2024)
- [Scene Understanding in Pick-and-Place Tasks: Analyzing Transformations Between Initial and Final Scenes](https://mlsanity.com/paper/2409.17720/) (2024)

## Abstract (arXiv)

Current NLP techniques have been greatly applied in different domains. In this paper, we propose a human-in-the-loop framework for robotic grasping in cluttered scenes, investigating a language interface to the grasping process, which allows the user to intervene by natural language commands. This framework is constructed on a state-of-the-art rasping baseline, where we substitute a scene-graph representation with a text representation of the scene using BERT. Experiments on both simulation and physical robot show that the proposed method outperforms conventional object-agnostic and scene-graph based methods in the literature. In addition, we find that with human intervention, performance can be significantly improved.