# DeVLBert: Learning Deconfounded Visio-Linguistic Representations

arXiv: https://arxiv.org/abs/2008.06884 · HTML: https://mlsanity.com/paper/2008.06884/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2020.

## Built on by (most cited first, 30 of 42)

- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning](https://mlsanity.com/paper/2203.13049/) (2022)
- [Vision-and-Language Navigation via Causal Learning](https://mlsanity.com/paper/2404.10241/) (2024)
- [One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models](https://mlsanity.com/paper/2406.05491/) (2024)
- [DocTr: Document Image Transformer for Geometric Unwarping and Illumination Correction](https://mlsanity.com/paper/2110.12942/) (2021)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval](https://mlsanity.com/paper/2304.03391/) (2023)
- [ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration](https://mlsanity.com/paper/2108.07073/) (2021)
- [Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models](https://mlsanity.com/paper/2207.01056/) (2022)
- [ComCLIP: Training-Free Compositional Image and Text Matching](https://mlsanity.com/paper/2211.13854/) (2022)
- [Intelligent Model Update Strategy for Sequential Recommendation](https://mlsanity.com/paper/2302.07335/) (2023)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralization](https://mlsanity.com/paper/2210.14562/) (2022)
- [End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Grounding](https://mlsanity.com/paper/2203.08013/) (2022)
- [Adaptive Hierarchical Graph Reasoning with Semantic Coherence for Video-and-Language Inference](https://mlsanity.com/paper/2107.12270/) (2021)
- [Domain-Specific Bias Filtering for Single Labeled Domain Generalization](https://mlsanity.com/paper/2110.00726/) (2021)
- [Causal Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2310.07379/) (2023)
- [AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model](https://mlsanity.com/paper/2308.07593/) (2023)
- [Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge](https://mlsanity.com/paper/2308.09311/) (2023)
- [Causal reasoning in typical computer vision tasks](https://mlsanity.com/paper/2307.13992/) (2023)
- [Why Do We Click: Visual Impression-aware News Recommendation](https://mlsanity.com/paper/2109.12651/) (2021)
- [Intelligent Request Strategy Design in Recommender System](https://mlsanity.com/paper/2206.12296/) (2022)
- [Dilated Context Integrated Network with Cross-Modal Consensus for Temporal Emotion Localization in Videos](https://mlsanity.com/paper/2208.01954/) (2022)
- [Click-through Rate Prediction with Auto-Quantized Contrastive Learning](https://mlsanity.com/paper/2109.13921/) (2021)
- [Fair Deepfake Detectors Can Generalize](https://mlsanity.com/paper/2507.02645/) (2025)
- [Towards Deconfounded Image-Text Matching with Causal Inference](https://mlsanity.com/paper/2408.12292/) (2024)
- [Learning Instrumental Variable from Data Fusion for Treatment Effect Estimation](https://mlsanity.com/paper/2208.10912/) (2022)
- [AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation](https://mlsanity.com/paper/2403.07030/) (2024)
- [FedCFA: Alleviating Simpson's Paradox in Model Aggregation with Counterfactual Federated Learning](https://mlsanity.com/paper/2412.18904/) (2024)

## Abstract (arXiv)

In this paper, we propose to investigate the problem of out-of-domain visio-linguistic pretraining, where the pretraining data distribution differs from that of downstream data on which the pretrained model will be fine-tuned. Existing methods for this problem are purely likelihood-based, leading to the spurious correlations and hurt the generalization ability when transferred to out-of-domain downstream tasks. By spurious correlation, we mean that the conditional probability of one token (object or word) given another one can be high (due to the dataset biases) without robust (causal) relationships between them. To mitigate such dataset biases, we propose a Deconfounded Visio-Linguistic Bert framework, abbreviated as DeVLBert, to perform intervention-based learning. We borrow the idea of the backdoor adjustment from the research field of causality and propose several neural-network based architectures for Bert-style out-of-domain pretraining. The quantitative results on three downstream tasks, Image Retrieval (IR), Zero-shot IR, and Visual Question Answering, show the effectiveness of DeVLBert by boosting generalization ability.