# Embodying Pre-Trained Word Embeddings Through Robot Actions

arXiv: https://arxiv.org/abs/2104.08521 · HTML: https://mlsanity.com/paper/2104.08521/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 17 Apr 2021.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://mlsanity.com/paper/1708.00524/) (2017)
- [Text2Action: Generative Adversarial Synthesis from Language to Action](https://mlsanity.com/paper/1710.05298/) (2017)
- [Multimodal Grounding for Language Processing](https://mlsanity.com/paper/1806.06371/) (2018)
- [Motion Switching with Sensory and Instruction Signals by designing Dynamical Systems using Deep Neural Network](https://mlsanity.com/paper/1712.05109/) (2017)
- [Exploiting Deep Semantics and Compositionality of Natural Language for Human-Robot-Interaction](https://mlsanity.com/paper/1604.06721/) (2016)
- [Word2vec to behavior: morphology facilitates the grounding of language in machines](https://mlsanity.com/paper/1908.01211/) (2019)

## Built on by (most cited first, 4 of 4)

- [Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model](https://mlsanity.com/paper/2308.15684/) (2023)
- [Learning Bidirectional Translation between Descriptions and Actions with Small Paired Data](https://mlsanity.com/paper/2203.04218/) (2022)
- [Ground Manipulator Primitive Tasks to Executable Actions using Large Language Models](https://mlsanity.com/paper/2308.06810/) (2023)
- [ProGAL-VLA: Grounded Alignment through Prospective Reasoning in Vision-Language-Action Models](https://mlsanity.com/paper/2604.09824/) (2026)

## Abstract (arXiv)

We propose a promising neural network model with which to acquire a grounded representation of robot actions and the linguistic descriptions thereof. Properly responding to various linguistic expressions, including polysemous words, is an important ability for robots that interact with people via linguistic dialogue. Previous studies have shown that robots can use words that are not included in the action-description paired datasets by using pre-trained word embeddings. However, the word embeddings trained under the distributional hypothesis are not grounded, as they are derived purely from a text corpus. In this letter, we transform the pre-trained word embeddings to embodied ones by using the robot's sensory-motor experiences. We extend a bidirectional translation model for actions and descriptions by incorporating non-linear layers that retrofit the word embeddings. By training the retrofit layer and the bidirectional translation model alternately, our proposed model is able to transform the pre-trained word embeddings to adapt to a paired action-description dataset. Our results demonstrate that the embeddings of synonyms form a semantic cluster by reflecting the experiences (actions and environments) of a robot. These embeddings allow the robot to properly generate actions from unseen words that are not paired with actions in a dataset.