Data to 7 Oct 2026

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

arXiv 1411.2539 ↗First posted 10 Nov 2014cs.LG cs.CL cs.CVcited by 197

Not on the map. Why?

Abstract (arXiv)

Inspired by recent advances in multimodal learning and machine translation, we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space. Our pipeline effectively unifies joint image-text embedding models with multimodal neural language models. We introduce the structure-content neural language model that disentangles the structure of a sentence to its content, conditioned on representations produced by the encoder. The encoder allows one to rank images and sentences while the decoder can generate novel descriptions from scratch. Using LSTM to encode sentences, we match the state-of-the-art performance on Flickr8K and Flickr30K without using object detections. We also set new best results when using the 19-layer Oxford convolutional network. Furthermore we show that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic e.g. *image of a blue car* - "blue" + "red" is near images of red cars. Sample captions generated for 800 images are made available for comparison.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Nov 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 59; 2022: 43; 2023: 46; 2024: 26; 2025: 15; 2026: 8.

197 papers.

Sort:

By area:

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
2021Combined Scaling for Zero-shot Transfer Learningcited by 182
2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
2023Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrievalcited by 131
2021Multimodal Contrastive Training for Visual Representation Learningcited by 122

38 more

2021Similarity Reasoning and Filtration for Image-Text Matchingcited by 121
2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
2023CLIP the Gap: A Single Domain Generalization Approach for Object Detectioncited by 87
2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
2021Cross Modal Retrieval with Querybank Normalisationcited by 71
2021DOC2PPT: Automatic Presentation Slides Generation from Scientific Documentscited by 68
2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 63
2023VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretrainingcited by 63
2023Concept Decomposition for Visual Exploration and Inspirationcited by 61
2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
2023AutoAD: Movie Description in Contextcited by 50
2021Towards General Purpose Vision Systemscited by 46
2021LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrievalcited by 44
2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
2021PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modelingcited by 43
2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
2022Text2Pos: Text-to-Point-Cloud Cross-Modal Localizationcited by 40
2023Improved Probabilistic Image-Text Representationscited by 40
2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 36
2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
2023Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrievalcited by 35
2022Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentorscited by 34
2022COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrievalcited by 33
2021HANet: Hierarchical Alignment Networks for Video-Text Retrievalcited by 31
2023A Large Cross-Modal Video Retrieval Dataset with Reading Comprehensioncited by 31
2023ViT-Lens: Towards Omni-modal Representationscited by 29
2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
2023Multi-modal Machine Learning in Engineering Design: A Review and Future Directionscited by 27
2021Step-Wise Hierarchical Alignment Network for Image-Text Matchingcited by 26
2021VLGrammar: Grounded Grammar Induction of Vision and Languagecited by 22
2023Cross-modal Active Complementary Learning with Self-refining Correspondencecited by 22

50 most cited of 197 All 197 (JSON)