Contrastive Visual-Linguistic Pretraining
Not on the map. Why?
Abstract (arXiv)
Several multi-modality representation learning approaches such as LXMERT and ViLBERT have been proposed recently. Such approaches can achieve superior performance due to the high-level semantic information captured during large-scale multimodal pretraining. However, as ViLBERT and LXMERT adopt visual region regression and classification loss, they often suffer from domain gap and noisy label problems, based on the visual features having been pretrained on the Visual Genome dataset. To overcome these issues, we propose unbiased Contrastive Visual-Linguistic Pretraining (CVLP), which constructs a visual self-supervised loss built upon contrastive learning. We evaluate CVLP on several down-stream tasks, including VQA, GQA and NLVR2 to validate the superiority of contrastive learning on multi-modality representation learning. Our code is available at: https://github.com/ArcherYunDong/CVLP-.
Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jul 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 8; 2022: 3; 2023: 2; 2024: 1.
14 papers.
Sort:
By area:
- 2021A Simple Long-Tailed Recognition Baseline via Vision-Language Modelcited by 49
- 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
- 2021Playing Lottery Tickets with Vision and Languagecited by 45
- 2021Language-Driven Image Style Transfercited by 36
- 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
- 2022Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detectioncited by 26
- 2021Multilingual Molecular Representation Learning via Contrastive Pre-trainingcited by 12
- 2021M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformerscited by 8
- 2021Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injectioncited by 8
- 2022Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answeringcited by 7
- 2023Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignmentcited by 5
- 2022Vision-and-Language Pretrainingcited by 2
- 2024Unified Video-Language Pre-training with Synchronized Audio12 May 2024
- 2023Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignment4 Dec 2023
- 2023CAVL: Learning Contrastive and Adaptive Representations of Vision and Language10 Apr 2023
- 2022Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering14 Sep 2022
- 2022Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection12 Jul 2022
- 2022Vision-and-Language Pretraining5 Jul 2022
- 2021Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injection13 Dec 2021
- 2021A Simple Long-Tailed Recognition Baseline via Vision-Language Model29 Nov 2021
- 2021Multilingual Molecular Representation Learning via Contrastive Pre-training18 Sep 2021
- 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions29 Jul 2021
- 2021Language-Driven Image Style Transfer1 Jun 2021
- 2021A Review on Explainability in Multimodal Deep Neural Nets17 May 2021
2 more
- 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
- 2021Playing Lottery Tickets with Vision and Languagecited by 45
- 2021Language-Driven Image Style Transfercited by 36
- 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
- 2022Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detectioncited by 26
- 2021Multilingual Molecular Representation Learning via Contrastive Pre-trainingcited by 12
- 2021M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformerscited by 8
- 2021Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injectioncited by 8
- 2023Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignmentcited by 5
- 2022Vision-and-Language Pretrainingcited by 2
- 2024Unified Video-Language Pre-training with Synchronized Audiocited by 2
- 2023CAVL: Learning Contrastive and Adaptive Representations of Vision and Languagecited by 0
- 2024Unified Video-Language Pre-training with Synchronized Audio12 May 2024
- 2023Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignment4 Dec 2023
- 2023CAVL: Learning Contrastive and Adaptive Representations of Vision and Language10 Apr 2023
- 2022Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection12 Jul 2022
- 2022Vision-and-Language Pretraining5 Jul 2022
- 2021Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injection13 Dec 2021
- 2021Multilingual Molecular Representation Learning via Contrastive Pre-training18 Sep 2021
- 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions29 Jul 2021
- 2021Language-Driven Image Style Transfer1 Jun 2021
- 2021A Review on Explainability in Multimodal Deep Neural Nets17 May 2021
- 2021Playing Lottery Tickets with Vision and Language23 Apr 2021
- 2021M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformers2 Apr 2021