mlsanityData to 7 Oct 2026

Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO

arXiv 2004.15020 ↗First posted 30 Apr 2020cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

By supporting multi-modal retrieval training and evaluation, image captioning datasets have spurred remarkable progress on representation learning. Unfortunately, datasets have limited cross-modal associations: images are not paired with other images, captions are only paired with other captions of the same image, there are no negative associations and there are missing positive cross-modal associations. This undermines research into how inter-modality learning impacts intra-modality tasks. We address this gap with Crisscrossed Captions (CxC), an extension of the MS-COCO dataset with human semantic similarity judgments for 267,095 intra- and inter-modality pairs. We report baseline results on CxC for strong existing unimodal and multimodal models. We also evaluate a multitask dual encoder trained on both image-caption and caption-caption pairs that crucially demonstrates CxC's value for measuring the influence of intra- and inter-modality learning.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 12; 2023: 10; 2024: 6; 2025: 7; 2026: 2.

46 papers.

Sort:

By area:

  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  3. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  4. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  5. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  6. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  7. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  8. 2023CompoDiff: Versatile Composed Image Retrieval With Latent Diffusioncited by 84
  9. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  10. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  11. 2021Adversarial Text-to-Image Synthesis: A Reviewcited by 52
  12. 2021MURAL: Multimodal, Multitask Retrieval Across Languagescited by 44
34 more
  1. 2021PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modelingcited by 43
  2. 2023Improved Probabilistic Image-Text Representationscited by 40
  3. 2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
  4. 2022Improving Cross-Modal Retrieval with Set of Diverse Embeddingscited by 33
  5. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  6. 2023EDIS: Entity-Driven Image Search over Multimodal Web Contentcited by 27
  7. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  8. 2022Words are all you need? Language as an approximation for human similarity judgmentscited by 18
  9. 2023Meaning Representations from Trajectories in Autoregressive Modelscited by 18
  10. 2023On the Difference of BERT-style and CLIP-style Text Encoderscited by 15
  11. 2021Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matchingcited by 13
  12. 2024Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrievalcited by 13
  13. 2022Predicting Human Similarity Judgments Using Large Language Modelscited by 10
  14. 2021Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Modelscited by 8
  15. 2022ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-trainingcited by 8
  16. 2023RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Modelscited by 7
  17. 2024Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encodingcited by 4
  18. 2024How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?cited by 4
  19. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4
  20. 2022Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrievalcited by 3
  21. 2022Uncertainty-based Cross-Modal Retrieval with Probabilistic Representationscited by 2
  22. 2023C-STS: Conditional Semantic Textual Similaritycited by 2
  23. 2025DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learningcited by 2
  24. 2022Image-Text Retrieval with Binary and Continuous Label Supervisioncited by 1
  25. 2024Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Modelscited by 1
  26. 2025Which Way from B to A: The role of embedding geometry in image interpolation for Stable Diffusioncited by 1
  27. 2025Advances and Challenges in Semantic Textual Similarity: A Comprehensive Surveycited by 1
  28. 2026Variational Adapter for Cross-modal Similarity Representationcited by 1
  29. 2022Revising Image-Text Retrieval via Multi-Modal Entailmentcited by 0
  30. 2023Refined Vision-Language Modeling for Fine-grained Multi-modal Pre-trainingcited by 0
  31. 2025Descriptive Image-Text Matching with Graded Contextual Similaritycited by 0
  32. 2025OS-HGAdapter: Open Semantic Hypergraph Adapter for Large Language Models Assisted Entropy-Enhanced Image-Text Alignmentcited by 0
  33. 2025AFRICAPTION: Establishing a New Paradigm for Image Captioning in African Languagescited by 0
  34. 2026Semantic Leakage from Image Embeddingscited by 0