mlsanityData to 7 Oct 2026

Multi-modal Alignment using Representation Codebook

arXiv 2203.00048 ↗First posted 28 Feb 2022cs.CV cs.AIcited by 63

Not on the map. Why?

Abstract (arXiv)

Aligning signals from different modalities is an important step in vision-language representation learning as it affects the performance of later stages such as cross-modality fusion. Since image and text typically reside in different regions of the feature space, directly aligning them at instance level is challenging especially when features are still evolving during training. In this paper, we propose to align at a higher and more stable level using cluster representation. Specifically, we treat image and text as two "views" of the same entity, and encode them into a joint vision-language coding space spanned by a dictionary of cluster centers (codebook). We contrast positive and negative samples via their cluster assignments while simultaneously optimizing the cluster centers. To further smooth out the learning process, we adopt a teacher-student distillation paradigm, where the momentum teacher of one view guides the student learning of the other. We evaluated our approach on common vision language benchmarks and obtain new SoTA on zero-shot cross modality retrieval while being competitive on various other transfer tasks.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 41 of its 51 references that are in the map. First posted on arXiv 28 Feb 2022.

Builds on

41 of 51 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
29 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  14. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  15. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  16. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  17. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  18. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  19. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  20. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  21. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  22. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  23. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  24. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  25. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  26. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  27. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  28. 2020Graph Optimal Transport for Cross-Domain Alignmentcited by 103
  29. 2020SLADE: A Self-Training Framework For Distance Metric Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 19; 2024: 16; 2025: 11; 2026: 9.

63 papers.

Sort:

By area:

  1. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  2. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  3. 2023Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Predictioncited by 86
  4. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 66
  5. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
  6. 2024Multimodal Prototyping for cancer survival predictioncited by 47
  7. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 24
  8. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  9. 2023Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Bankscited by 20
  10. 2024Concept-Guided Prompt Learning for Generalization in Vision-Language Modelscited by 20
  11. 2022Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignmentcited by 19
  12. 2023Learning to Adapt CLIP for Few-Shot Monocular Depth Estimationcited by 19
38 more
  1. 2023VaQuitA: Enhancing Alignment in LLM-Assisted Video Understandingcited by 17
  2. 2023Domain Aligned CLIP for Few-shot Classificationcited by 16
  3. 2024Law of Vision Representation in MLLMscited by 16
  4. 2024X-Former: Unifying Contrastive and Reconstruction Learning for MLLMscited by 14
  5. 2026Learning Continuous Wasserstein Barycenter Space for Generalized All-in-One Image Restorationcited by 14
  6. 2023Cross-Modal Concept Learning and Inference for Vision-Language Modelscited by 12
  7. 2025VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Setcited by 12
  8. 2022MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learningcited by 11
  9. 2024Bridging Vision and Language Spaces with Assignment Predictioncited by 11
  10. 2025Bridging Domain Generalization to Multimodal Domain Generalization via Unified Representationscited by 9
  11. 2024Unified Lexical Representation for Interpretable Visual-Language Alignmentcited by 8
  12. 2023Masked Autoencoding Does Not Help Natural Language Supervision at Scalecited by 7
  13. 2024PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adaptercited by 7
  14. 2024Conceptual Codebook Learning for Vision-Language Modelscited by 7
  15. 2022MHMS: Multimodal Hierarchical Multimedia Summarizationcited by 6
  16. 2022Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignmentcited by 6
  17. 2023Mobile Image Restoration via Prior Quantizationcited by 6
  18. 2023BDC-Adapter: Brownian Distance Covariance for Better Vision-Language Reasoningcited by 6
  19. 2023Multi-Modal Representation Learning with Text-Driven Soft Maskscited by 5
  20. 2023Unsupervised Prototype Adapter for Vision-Language Modelscited by 5
  21. 2024Semantic Residual for Multimodal Unified Discrete Representationcited by 5
  22. 2023Deep Equilibrium Multimodal Fusioncited by 4
  23. 2024From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signalscited by 4
  24. 2024Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representationscited by 4
  25. 2025Open-set Cross Modal Generalization via Multimodal Unified Representationcited by 4
  26. 2023MoMo: A shared encoder Model for text, image and multi-Modal representationscited by 3
  27. 2024Data Processing Techniques for Modern Multimodal Modelscited by 3
  28. 2024WeCromCL: Weakly Supervised Cross-Modality Contrastive Learning for Transcription-only Supervised Text Spottingcited by 3
  29. 2024Aligning Large Language Models and Geometric Deep Models for Protein Representationcited by 3
  30. 2025BaryIR: Learning Multi-Source Unified Representation in Continuous Barycenter Space for Generalizable All-in-One Image Restorationcited by 3
  31. 2023SimCon Loss with Multiple Views for Text Supervised Semantic Segmentationcited by 2
  32. 2023TOT: Topology-Aware Optimal Transport For Multimodal Hate Detectioncited by 2
  33. 2023A Reminder of its Brittleness: Language Reward Shaping May Hinder Learning for Instruction Following Agentscited by 2
  34. 2025COST: Contrastive One-Stage Transformer for Vision-Language Small Object Trackingcited by 2
  35. 2024COT: A Generative Approach for Hate Speech Counter-Narratives via Contrastive Optimal Transportcited by 1
  36. 2025Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approachcited by 1
  37. 2025Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcarecited by 1
  38. 2026Decoupled Hierarchical Distillation for Multimodal Emotion Recognitioncited by 1

50 most cited of 63 All 63 (JSON)