mlsanityData to 14 Oct 2026

Learning Deep Structure-Preserving Image-Text Embeddings

arXiv 1511.06078 ↗First posted 19 Nov 2015cs.CV cs.CL cs.LGcited by 139

Not on the map. Why?

Abstract (arXiv)

This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a large margin objective that combines cross-view ranking constraints with within-view neighborhood structure preservation constraints inspired by metric learning literature. Extensive experiments show that our approach gains significant improvements in accuracy for image-to-text and text-to-image retrieval. Our method achieves new state-of-the-art results on the Flickr30K and MSCOCO image-sentence datasets and shows promise on the new task of phrase localization on the Flickr30K Entities dataset.

Key facts. Cited by 139 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 50; 2022: 41; 2023: 20; 2024: 18; 2025: 8; 2026: 2.

139 papers.

Sort:

By area:

  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  2. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  3. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  4. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  5. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  6. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 164
  7. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 151
  8. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130
  9. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 128
  10. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 124
  11. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  12. 2021Similarity Reasoning and Filtration for Image-Text Matchingcited by 121
38 more
  1. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  2. 2021Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identificationcited by 110
  3. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  4. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  5. 2021Negative Sample Matters: A Renaissance of Metric Learning for Temporal Groundingcited by 90
  6. 2022ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similaritycited by 83
  7. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  8. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  9. 2021CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representationscited by 72
  10. 2021Cross Modal Retrieval with Querybank Normalisationcited by 72
  11. 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 64
  12. 2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62
  13. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  14. 2022TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformercited by 42
  15. 2022Text2Pos: Text-to-Point-Cloud Cross-Modal Localizationcited by 40
  16. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 36
  17. 2021Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Imagescited by 33
  18. 2022COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrievalcited by 33
  19. 2021Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detectioncited by 32
  20. 2021Multimodal Entity Linking for Tweetscited by 28
  21. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  22. 2021A Survey on Multi-modal Summarizationcited by 26
  23. 2022Progressive Learning for Image Retrieval with Hybrid-Modality Queriescited by 26
  24. 2023Noisy Correspondence Learning with Meta Similarity Correctioncited by 26
  25. 2024Composed Video Retrieval via Enriched Context and Discriminative Embeddingscited by 26
  26. 2022Video-Text Representation Learning via Differentiable Weak Temporal Alignmentcited by 24
  27. 2022DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Groundingcited by 24
  28. 2021Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrievalcited by 23
  29. 2023ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embeddingcited by 23
  30. 2022Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 22
  31. 2021Embedding Arithmetic of Multimodal Queries for Image Retrievalcited by 21
  32. 2022Adapting CLIP For Phrase Localization Without Further Trainingcited by 21
  33. 2021Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learningcited by 20
  34. 2021Detector-Free Weakly Supervised Grounding by Separationcited by 16
  35. 2022Learning Program Representations for Food Images and Cooking Recipescited by 16
  36. 2023Deep Learning for Video-Text Retrieval: a Reviewcited by 15
  37. 2024HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modelingcited by 15
  38. 2022Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrievalcited by 14

50 most cited of 139 All 139 (JSON)