# Grounding and Distinguishing Conceptual Vocabulary Through Similarity Learning in Embodied Simulations

arXiv: https://arxiv.org/abs/2305.13668 · HTML: https://mlsanity.com/paper/2305.13668/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 46 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Grounded Language Learning in a Simulated 3D World](https://mlsanity.com/paper/1706.06551/) (2017)
- [Learning to Make Analogies by Contrasting Abstract Relational Structure](https://mlsanity.com/paper/1902.00120/) (2019)
- [Language Grounding with 3D Objects](https://mlsanity.com/paper/2107.12514/) (2021)
- [Multimodal Joint Attribute Prediction and Value Extraction for E-commerce Product](https://mlsanity.com/paper/2009.07162/) (2020)
- [Combining Language and Vision with a Multimodal Skip-gram Model](https://mlsanity.com/paper/1501.02598/) (2015)
- [Multi-Label Contrastive Learning for Abstract Visual Reasoning](https://mlsanity.com/paper/2012.01944/) (2020)
- [Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation](https://mlsanity.com/paper/2007.00229/) (2020)
- [Emergent Discrete Communication in Semantic Spaces](https://mlsanity.com/paper/2108.01828/) (2021)
- [Towards Human-Agent Communication via the Information Bottleneck Principle](https://mlsanity.com/paper/2207.00088/) (2022)
- [Pretraining on Interactions for Learning Grounded Affordance Representations](https://mlsanity.com/paper/2207.02272/) (2022)
- [Canonical Face Embeddings](https://mlsanity.com/paper/2106.07822/) (2021)
- [Detecting and Accommodating Novel Types and Concepts in an Embodied Simulation Environment](https://mlsanity.com/paper/2211.04555/) (2022)
- [Do Trajectories Encode Verb Meaning?](https://mlsanity.com/paper/2206.11953/) (2022)

## Built on by (most cited first, 6 of 6)

- [From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models](https://mlsanity.com/paper/2411.05036/) (2024)
- [Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles](https://mlsanity.com/paper/2404.08949/) (2024)
- [Grounded and Well-rounded: A Methodological Approach to the Study of Cross-modal and Cross-lingual Grounding](https://mlsanity.com/paper/2310.11938/) (2023)
- [Combining Automatic Coding and Instructor Input to Generate ENA Visualizations for Asynchronous Online Discussion](https://mlsanity.com/paper/2308.13549/) (2023)
- [Language, Environment, and Robotic Navigation](https://mlsanity.com/paper/2404.03049/) (2024)
- [Computational Thought Experiments for a More Rigorous Philosophy and Science of the Mind](https://mlsanity.com/paper/2405.08304/) (2024)

## Abstract (arXiv)

We present a novel method for using agent experiences gathered through an embodied simulation to ground contextualized word vectors to object representations. We use similarity learning to make comparisons between different object types based on their properties when interacted with, and to extract common features pertaining to the objects' behavior. We then use an affine transformation to calculate a projection matrix that transforms contextualized word vectors from different transformer-based language models into this learned space, and evaluate whether new test instances of transformed token vectors identify the correct concept in the object embedding space. Our results expose properties of the embedding spaces of four different transformer models and show that grounding object token vectors is usually more helpful to grounding verb and attribute token vectors than the reverse, which reflects earlier conclusions in the analogical reasoning and psycholinguistic literature.