mlsanityData to 7 Oct 2026

Learning Visually Grounded Sentence Representations

arXiv 1707.06320 ↗First posted 19 Jul 2017cs.CL cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

We introduce a variety of models, trained on a supervised image captioning corpus to predict the image features for a given caption, to perform sentence representation grounding. We train a grounded sentence encoder that achieves good performance on COCO caption and image retrieval and subsequently show that this encoder can successfully be transferred to various NLP tasks, with improved performance over text-only models. Lastly, we analyze the contribution of grounding, and show that word embeddings learned by this system outperform non-grounded ones.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 12; 2023: 3; 2024: 2; 2025: 1.

24 papers.

Sort:

By area:

  1. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  2. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  3. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  4. 2022VALHALLA: Visual Hallucination for Machine Translationcited by 27
  5. 2021VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfercited by 22
  6. 2022MCSE: Multimodal Contrastive Learning of Sentence Embeddingscited by 22
  7. 2022UNIMO-2: End-to-End Unified Vision-Language Grounded Learningcited by 19
  8. 2022Imagination-Augmented Natural Language Understandingcited by 19
  9. 2023Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understandingcited by 19
  10. 2021Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?cited by 18
  11. 2022Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imaginationcited by 18
  12. 2024Supervised Fine-tuning in turn Improves Visual Foundation Modelscited by 13
12 more
  1. 2024KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learningcited by 13
  2. 2023Towards More Human-like AI Communication: A Review of Emergent Communication Researchcited by 12
  3. 2021Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Trainingcited by 8
  4. 2021Visual Grounding Strategies for Text-Only Natural Language Processingcited by 7
  5. 2021Semantic sentence similarity: size does not always mattercited by 5
  6. 2022Language with Vision: a Study on Grounded Word and Sentence Embeddingscited by 5
  7. 2023Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignmentcited by 5
  8. 2022Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledgecited by 2
  9. 2022Do Trajectories Encode Verb Meaning?cited by 2
  10. 2025FLUID: Flow-Latent Unified Integration via Token Distillation for Expert Specialization in Multimodal Learningcited by 1
  11. 2022How direct is the link between words and images?cited by 0
  12. 2022Like a bilingual baby: The advantage of visually grounding a bilingual language modelcited by 0