# Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training

arXiv: https://arxiv.org/abs/2104.07500 · HTML: https://mlsanity.com/paper/2104.07500/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 65 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Learning Fine-grained Image Similarity with Deep Ranking](https://mlsanity.com/paper/1404.4661/) (2014)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision](https://mlsanity.com/paper/2010.06775/) (2020)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://mlsanity.com/paper/1608.00869/) (2016)
- [Evaluating Word Embedding Models: Methods and Experimental Results](https://mlsanity.com/paper/1901.09785/) (2019)
- [Combining Language and Vision with a Multimodal Skip-gram Model](https://mlsanity.com/paper/1501.02598/) (2015)
- [Evaluation of sentence embeddings in downstream and linguistic probing tasks](https://mlsanity.com/paper/1806.06259/) (2018)
- [Learning Visually Grounded Sentence Representations](https://mlsanity.com/paper/1707.06320/) (2017)
- [Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations](https://mlsanity.com/paper/1803.01400/) (2018)
- [Incorporating Visual Semantics into Sentence Representations within a Grounded Space](https://mlsanity.com/paper/2002.02734/) (2020)
- [Learning language through pictures](https://mlsanity.com/paper/1506.03694/) (2015)
- [Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes](https://mlsanity.com/paper/1511.07067/) (2015)
- [Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images](https://mlsanity.com/paper/1611.08321/) (2016)
- [Learning Multi-Modal Word Representation Grounded in Visual Context](https://mlsanity.com/paper/1711.03483/) (2017)
- [Visual Grounding Strategies for Text-Only Natural Language Processing](https://mlsanity.com/paper/2103.13942/) (2021)
- [Language Features Matter: Effective Language Representations for Vision-Language Tasks](https://mlsanity.com/paper/1908.06327/) (2019)
- [Better Text Understanding Through Image-To-Text Transfer](https://mlsanity.com/paper/1705.08386/) (2017)

## Built on by (most cited first, 8 of 8)

- [Modeling morphology with Linear Discriminative Learning: considerations and design choices](https://mlsanity.com/paper/2106.07936/) (2021)
- [Language with Vision: a Study on Grounded Word and Sentence Embeddings](https://mlsanity.com/paper/2206.08823/) (2022)
- [How trial-to-trial learning shapes mappings in the mental lexicon: Modelling Lexical Decision with Linear Discriminative Learning](https://mlsanity.com/paper/2207.00430/) (2022)
- [Is deeper always better? Replacing linear mappings with deep learning networks in the Discriminative Lexicon Model](https://mlsanity.com/paper/2410.04259/) (2024)
- [Semantic properties of English nominal pluralization: Insights from word embeddings](https://mlsanity.com/paper/2203.15424/) (2022)
- [Making sense of spoken plurals](https://mlsanity.com/paper/2207.01947/) (2022)
- [How direct is the link between words and images?](https://mlsanity.com/paper/2206.15381/) (2022)
- [Visual Grounding of Inter-lingual Word-Embeddings](https://mlsanity.com/paper/2209.03714/) (2022)

## Abstract (arXiv)

Language grounding aims at linking the symbolic representation of language (e.g., words) into the rich perceptual knowledge of the outside world. The general approach is to embed both textual and visual information into a common space -the grounded space-confined by an explicit relationship between both modalities. We argue that this approach sacrifices the abstract knowledge obtained from linguistic co-occurrence statistics in the process of acquiring perceptual information. The focus of this paper is to solve this issue by implicitly grounding the word embeddings. Rather than learning two mappings into a joint space, our approach integrates modalities by determining a reversible grounded mapping between the textual and the grounded space by means of multi-task learning. Evaluations on intrinsic and extrinsic tasks show that our embeddings are highly beneficial for both abstract and concrete words. They are strongly correlated with human judgments and outperform previous works on a wide range of benchmarks. Our grounded embeddings are publicly available here.