# Natural Language Instructions for Intuitive Human Interaction with Robotic Assistants in Field Construction Work

arXiv: https://arxiv.org/abs/2307.04195 · HTML: https://mlsanity.com/paper/2307.04195/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 131 references that are in the map.
- First posted on arXiv 9 Jul 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions](https://mlsanity.com/paper/1710.06280/) (2017)
- [A Comparison of LSTM and BERT for Small Corpus](https://mlsanity.com/paper/2009.05451/) (2020)
- [Improved Trust in Human-Robot Collaboration with ChatGPT](https://mlsanity.com/paper/2304.12529/) (2023)
- [Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks](https://mlsanity.com/paper/1707.06799/) (2017)
- [Understanding Natural Language Instructions for Fetching Daily Objects Using GAN-Based Multimodal Target-Source Classification](https://mlsanity.com/paper/1906.06830/) (2019)
- [Learning Object Placements For Relational Instructions by Hallucinating Scene Representations](https://mlsanity.com/paper/2001.08481/) (2020)
- [Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning](https://mlsanity.com/paper/1904.12907/) (2019)
- [Pretrained Domain-Specific Language Model for General Information Retrieval Tasks in the AEC Domain](https://mlsanity.com/paper/2203.04729/) (2022)
- [Empirical Study of Deep Learning for Text Classification in Legal Document Review](https://mlsanity.com/paper/1904.01723/) (2019)
- [Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robots](https://mlsanity.com/paper/2107.00811/) (2021)
- [Exploiting Deep Semantics and Compositionality of Natural Language for Human-Robot-Interaction](https://mlsanity.com/paper/1604.06721/) (2016)
- [Systems of natural-language-facilitated human-robot cooperation: A review](https://mlsanity.com/paper/1701.08269/) (2017)
- [Augmented Reality for Maintenance Tasks with ChatGPT for Automated Text-to-Action](https://mlsanity.com/paper/2307.03351/) (2023)
- [Recognizing Chinese Judicial Named Entity using BiLSTM-CRF](https://mlsanity.com/paper/2006.00464/) (2020)

## Built on by (most cited first, 3 of 3)

- [AnywhereVLA: Language-Conditioned Exploration and Mobile Manipulation](https://mlsanity.com/paper/2509.21006/) (2025)
- [Advances in Transformers for Robotic Applications: A Review](https://mlsanity.com/paper/2412.10599/) (2024)
- [Lang2Lift: A Language-Guided Autonomous Forklift System for Outdoor Industrial Pallet Handling](https://mlsanity.com/paper/2508.15427/) (2025)

## Abstract (arXiv)

The introduction of robots is widely considered to have significant potential of alleviating the issues of worker shortage and stagnant productivity that afflict the construction industry. However, it is challenging to use fully automated robots in complex and unstructured construction sites. Human-Robot Collaboration (HRC) has shown promise of combining human workers' flexibility and robot assistants' physical abilities to jointly address the uncertainties inherent in construction work. When introducing HRC in construction, it is critical to recognize the importance of teamwork and supervision in field construction and establish a natural and intuitive communication system for the human workers and robotic assistants. Natural language-based interaction can enable intuitive and familiar communication with robots for human workers who are non-experts in robot programming. However, limited research has been conducted on this topic in construction. This paper proposes a framework to allow human workers to interact with construction robots based on natural language instructions. The proposed method consists of three stages: Natural Language Understanding (NLU), Information Mapping (IM), and Robot Control (RC). Natural language instructions are input to a language model to predict a tag for each word in the NLU module. The IM module uses the result of the NLU module and building component information to generate the final instructional output essential for a robot to acknowledge and perform the construction task. A case study for drywall installation is conducted to evaluate the proposed approach. The obtained results highlight the potential of using natural language-based interaction to replicate the communication that occurs between human workers within the context of human-robot teams.