# Contrastive Visual-Linguistic Pretraining

arXiv: https://arxiv.org/abs/2007.13135 · HTML: https://mlsanity.com/paper/2007.13135/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2020.

## Built on by (most cited first, 14 of 14)

- [A Simple Long-Tailed Recognition Baseline via Vision-Language Model](https://mlsanity.com/paper/2111.14745/) (2021)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Playing Lottery Tickets with Vision and Language](https://mlsanity.com/paper/2104.11832/) (2021)
- [Language-Driven Image Style Transfer](https://mlsanity.com/paper/2106.00178/) (2021)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection](https://mlsanity.com/paper/2207.05500/) (2022)
- [Multilingual Molecular Representation Learning via Contrastive Pre-training](https://mlsanity.com/paper/2109.08830/) (2021)
- [M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformers](https://mlsanity.com/paper/2104.01122/) (2021)
- [Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injection](https://mlsanity.com/paper/2112.06888/) (2021)
- [Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering](https://mlsanity.com/paper/2209.06954/) (2022)
- [Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignment](https://mlsanity.com/paper/2312.01592/) (2023)
- [Vision-and-Language Pretraining](https://mlsanity.com/paper/2207.01772/) (2022)
- [Unified Video-Language Pre-training with Synchronized Audio](https://mlsanity.com/paper/2405.07202/) (2024)
- [CAVL: Learning Contrastive and Adaptive Representations of Vision and Language](https://mlsanity.com/paper/2304.04399/) (2023)

## Abstract (arXiv)

Several multi-modality representation learning approaches such as LXMERT and ViLBERT have been proposed recently. Such approaches can achieve superior performance due to the high-level semantic information captured during large-scale multimodal pretraining. However, as ViLBERT and LXMERT adopt visual region regression and classification loss, they often suffer from domain gap and noisy label problems, based on the visual features having been pretrained on the Visual Genome dataset. To overcome these issues, we propose unbiased Contrastive Visual-Linguistic Pretraining (CVLP), which constructs a visual self-supervised loss built upon contrastive learning. We evaluate CVLP on several down-stream tasks, including VQA, GQA and NLVR2 to validate the superiority of contrastive learning on multi-modality representation learning. Our code is available at: https://github.com/ArcherYunDong/CVLP-.