# Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction

arXiv: https://arxiv.org/abs/2104.08874 · HTML: https://mlsanity.com/paper/2104.08874/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks](https://mlsanity.com/paper/1810.00825/) (2018)
- [Embedding Logical Queries on Knowledge Graphs](https://mlsanity.com/paper/1806.01445/) (2018)
- [Grounded Language Learning Fast and Slow](https://mlsanity.com/paper/2009.01719/) (2020)
- [Meta-Prod2Vec - Product Embeddings Using Side-Information for Recommendation](https://mlsanity.com/paper/1607.07326/) (2016)
- [E-commerce in Your Inbox: Product Recommendations at Scale](https://mlsanity.com/paper/1606.07154/) (2016)
- [Entity-aware Image Caption Generation](https://mlsanity.com/paper/1804.07889/) (2018)
- [Interactive Grounded Language Acquisition and Generalization in a 2D World](https://mlsanity.com/paper/1802.01433/) (2018)
- [Fantastic Embeddings and How to Align Them: Zero-Shot Inference in a Multi-Shop Scenario](https://mlsanity.com/paper/2007.14906/) (2020)
- [Query2Prod2Vec Grounded Word Embeddings for eCommerce](https://mlsanity.com/paper/2104.02061/) (2021)
- [How to Grow a (Product) Tree: Personalized Category Suggestions for eCommerce Type-Ahead](https://mlsanity.com/paper/2005.12781/) (2020)
- [Exploring Exploration: Comparing Children with RL Agents in Unified Environments](https://mlsanity.com/paper/2005.02880/) (2020)
- [Revisiting Skip-Gram Negative Sampling Model with Rectification](https://mlsanity.com/paper/1804.00306/) (2018)
- ["An Image is Worth a Thousand Features": Scalable Product Representations for In-Session Type-Ahead Personalization](https://mlsanity.com/paper/2003.07160/) (2020)
- [Interpretable Set Functions](https://mlsanity.com/paper/1806.00050/) (2018)
- [Lexical Learning as an Online Optimal Experiment: Building Efficient Search Engines through Human-Machine Collaboration](https://mlsanity.com/paper/1910.14164/) (2019)
- [On The Plurality of Graphs](https://mlsanity.com/paper/2008.00920/) (2020)

## Built on by (most cited first, 5 of 5)

- [Contrastive language and vision learning of general fashion concepts](https://mlsanity.com/paper/2204.03972/) (2022)
- [Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations](https://mlsanity.com/paper/2405.13828/) (2024)
- ["Does it come in black?" CLIP-like models are zero-shot recommenders](https://mlsanity.com/paper/2204.02473/) (2022)
- [Witgenstein's influence on artificial intelligence](https://mlsanity.com/paper/2302.01570/) (2023)
- [(Vector) Space is Not the Final Frontier: Product Search as Program Synthesis](https://mlsanity.com/paper/2304.11473/) (2023)

## Abstract (arXiv)

We investigate grounded language learning through real-world data, by modelling a teacher-learner dynamics through the natural interactions occurring between users and search engines; in particular, we explore the emergence of semantic generalization from unsupervised dense representations outside of synthetic environments. A grounding domain, a denotation function and a composition function are learned from user data only. We show how the resulting semantics for noun phrases exhibits compositional properties while being fully learnable without any explicit labelling. We benchmark our grounded semantics on compositionality and zero-shot inference tasks, and we show that it provides better results and better generalizations than SOTA non-grounded models, such as word2vec and BERT.