mlsanityData to 7 Oct 2026

Equivariant Similarity for Vision-Language Foundation Models

arXiv 2303.14465 ↗First posted 25 Mar 2023cs.CVcited by 51

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This study explores the concept of equivariance in vision-language foundation models (VLMs), focusing specifically on the multimodal similarity function that is not only the major training objective but also the core delivery to support downstream tasks. Unlike the existing image-text similarity objective which only categorizes matched pairs as similar and unmatched pairs as dissimilar, equivariance also requires similarity to vary faithfully according to the semantic changes. This allows VLMs to generalize better to nuanced and unseen multimodal compositions. However, modeling equivariance is challenging as the ground truth of semantic change is difficult to collect. For example, given an image-text pair about a dog, it is unclear to what extent the similarity changes when the pixel is changed from dog to cat? To this end, we propose EqSim, a regularization loss that can be efficiently calculated from any two matched training pairs and easily pluggable into existing image-text retrieval fine-tuning. Meanwhile, to further diagnose the equivariance of VLMs, we present a new challenging benchmark EqBen. Compared to the existing evaluation sets, EqBen is the first to focus on "visual-minimal change". Extensive experiments show the lack of equivariance in current VLMs and validate the effectiveness of EqSim. Code is available at https://github.com/Wangt-CN/EqBen.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 83 references that are in the map. First posted on arXiv 25 Mar 2023.

Builds on

64 of 83 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2015VQA: Visual Question Answeringcited by 2,489
38 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  14. 2016Group Equivariant Convolutional Networkscited by 1,154
  15. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  16. 2016Modeling Context in Referring Expressionscited by 1,054
  17. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  18. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  19. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  20. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  21. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  22. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  23. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  24. 2018Spherical CNNscited by 452
  25. 2019General E(2)-Equivariant Steerable CNNscited by 441
  26. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  27. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  28. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  29. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  30. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  31. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  32. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  33. 2019RUBi: Reducing Unimodal Biases in Visual Question Answeringcited by 183
  34. 2020Support-set bottlenecks for video-text representation learningcited by 170
  35. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  36. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  37. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  38. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 25; 2025: 11; 2026: 8.

51 papers.

Sort:

By area:

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  2. 2024An Introduction to Vision-Language Modelingcited by 175
  3. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  4. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  5. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  6. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  7. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  8. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  9. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  10. 2023PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learningcited by 31
  11. 2024Auto-Encoding Morph-Tokens for Multimodal LLMcited by 30
  12. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
38 more
  1. 2023Insect-Foundation: A Foundation Model and Large-scale 1M Dataset for Visual Insect Understandingcited by 29
  2. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  3. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  4. 2024VisMin: Visual Minimal-Change Understandingcited by 23
  5. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  6. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decompositioncited by 16
  7. 2024TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Modelscited by 12
  8. 2024QClusformer: A Quantum Transformer-based Framework for Unsupervised Visual Clusteringcited by 11
  9. 2024Progressive Compositionality in Text-to-Image Generative Modelscited by 11
  10. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
  11. 2025VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Modelscited by 11
  12. 2024Quantum Visual Feature Encoding Revisitedcited by 10
  13. 2024RankCLIP: Ranking-Consistent Language-Image Pretrainingcited by 9
  14. 2023Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistencycited by 8
  15. 2024Comparison Visual Instruction Tuningcited by 8
  16. 2024Natural Language Inference Improves Compositionality in Vision-Language Modelscited by 8
  17. 2024Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic Viewcited by 7
  18. 2025Diffusion Classifiers Understand Compositionality, but Conditions Applycited by 7
  19. 2024BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrievalcited by 6
  20. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarkscited by 5
  21. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captionscited by 5
  22. 2024Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspectivecited by 4
  23. 2026SpatialReward: Verifiable Spatial Reward Modeling for Fine-Grained Spatial Consistency in Text-to-Image Generationcited by 4
  24. 2023Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRIcited by 3
  25. 2025Multimodal Language Models as Text-to-Image Model Evaluatorscited by 3
  26. 2024ED-SAM: An Efficient Diffusion Sampling Approach to Domain Generalization in Vision-Language Foundation Modelscited by 2
  27. 2024Exploring the Spectrum of Visio-Linguistic Compositionality and Recognitioncited by 2
  28. 2025SCRAMBLe : Enhancing Multimodal LLM Compositionality with Synthetic Preference Datacited by 2
  29. 2025Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptionscited by 2
  30. 2025SAP-Bench: Benchmarking Multimodal Large Language Models in Surgical Action Planningcited by 1
  31. 2025Adding simple structure at inference improves Vision-Language Compositionalitycited by 1
  32. 2026Multimodal Language Models Cannot Spot Spatial Inconsistenciescited by 1
  33. 2026Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimizationcited by 1
  34. 2025Talk in Pieces, See in Whole: Disentangled and Hierarchical Representation Learning in Language-based Object Detectioncited by 0
  35. 2026Explicit Logic Channel for Validation and Enhancement of MLLMs on Zero-Shot Taskscited by 0
  36. 2026FineGen: A VLM-based Multi-Agent Framework for Fine-Grained Image-Text Dataset Constructioncited by 0
  37. 2026MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representationscited by 0
  38. 2026SNIP++: Fine-Grained Symbolic-Numerical Alignment for Symbolic Regressioncited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations
  2. 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
  3. 2026Variational Adapter for Cross-modal Similarity Representation
  4. 2024CAST: Cross-modal Alignment Similarity Test for Vision Language Models
  5. 2025Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
  6. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?
  7. 2026Indexing Multimodal Language Models for Large-scale Image Retrieval
  8. 2026LoMo: Local Modality Substitution for Deeper Vision-Language Fusion