mlsanityData to 7 Oct 2026

Deep Fragment Embeddings for Bidirectional Image Sentence Mapping

arXiv 1406.5679 ↗First posted 22 Jun 2014cs.CV cs.CL cs.LGcited by 99

Not on the map. Why?

Abstract (arXiv)

We introduce a model for bidirectional retrieval of images and sentences through a multi-modal embedding of visual and natural language data. Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space. In addition to a ranking objective seen in previous work, this allows us to add a new fragment alignment objective that learns to directly associate these fragments across modalities. Extensive experimental evaluation shows that reasoning on both the global level of images and sentences and the finer level of their respective fragments significantly improves performance on image-sentence retrieval tasks. Additionally, our model provides interpretable predictions since the inferred inter-modal fragment alignment is explicit.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jun 2014.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 31; 2022: 27; 2023: 25; 2024: 11; 2025: 5.

99 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  4. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  5. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  6. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  7. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 150
  8. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  9. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  10. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  11. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  12. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
38 more
  1. 2023Concept Decomposition for Visual Exploration and Inspirationcited by 61
  2. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
  3. 2023Towards General Visual-Linguistic Face Forgery Detectioncited by 45
  4. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  5. 2021Visual Goal-Step Inference using wikiHowcited by 42
  6. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37
  7. 2021Relation-aware Instance Refinement for Weakly Supervised Visual Groundingcited by 35
  8. 2021Dual-Key Multimodal Backdoors for Visual Question Answeringcited by 35
  9. 2021Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillationcited by 35
  10. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
  11. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captionscited by 33
  12. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  13. 2021Linguistic Structures as Weak Supervision for Visual Scene Graph Generationcited by 29
  14. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  15. 2023Multi-modal Machine Learning in Engineering Design: A Review and Future Directionscited by 27
  16. 2021A Survey on Multi-modal Summarizationcited by 26
  17. 2022DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Groundingcited by 24
  18. 2021A Joint Network for Grasp Detection Conditioned on Natural Language Commandscited by 23
  19. 2021Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrievalcited by 23
  20. 2021Masking Modalities for Cross-modal Video Retrievalcited by 23
  21. 2021VLGrammar: Grounded Grammar Induction of Vision and Languagecited by 22
  22. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  23. 2021Joint Multimedia Event Extraction from Video and Articlecited by 19
  24. 2021Deconfounded Visual Groundingcited by 16
  25. 2022Learning Program Representations for Food Images and Cooking Recipescited by 16
  26. 2021Parts2Words: Learning Joint Embedding of Point Clouds and Texts by Bidirectional Matching between Parts and Wordscited by 14
  27. 2022Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrievalcited by 14
  28. 2023Visual Grounding of Whole Radiology Reports for 3D CT Imagescited by 13
  29. 2021Exploring Visual Engagement Signals for Representation Learningcited by 12
  30. 2022PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Groundingcited by 12
  31. 2023Object Recognition as Next Token Predictioncited by 12
  32. 2021Neural Attention for Image Captioning: Review of Outstanding Methodscited by 11
  33. 2023OPORP: One Permutation + One Random Projectioncited by 11
  34. 2022MDMMT-2: Multidomain Multimodal Transformer for Video Retrieval, One More Step Towards Generalizationcited by 10
  35. 2022Audio-text Retrieval in Contextcited by 10
  36. 2023Predictive Chemistry Augmented with Text Retrievalcited by 9
  37. 2025ELIP: Enhanced Visual-Language Foundation Models for Image Retrievalcited by 9
  38. 2021Evaluation of Audio-Visual Alignments in Visually Grounded Speech Modelscited by 8

50 most cited of 99 All 99 (JSON)