# Seeking the Sufficiency and Necessity Causal Features in Multimodal Representation Learning

arXiv: https://arxiv.org/abs/2408.16577 · HTML: https://mlsanity.com/paper/2408.16577/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 48 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 30 of 35)

- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [DAGs with NO TEARS: Continuous Optimization for Structure Learning](https://mlsanity.com/paper/1803.01422/) (2018)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [DreamLLM: Synergistic Multimodal Comprehension and Creation](https://mlsanity.com/paper/2309.11499/) (2023)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [Causal Discovery with Continuous Additive Noise Models](https://mlsanity.com/paper/1309.6779/) (2013)
- [Invariant Risk Minimization Games](https://mlsanity.com/paper/2002.04692/) (2020)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Causal Discovery from Heterogeneous/Nonstationary Data with Independent Changes](https://mlsanity.com/paper/1903.01672/) (2019)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Causal Discovery with Reinforcement Learning](https://mlsanity.com/paper/1906.04477/) (2019)
- [Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs](https://mlsanity.com/paper/2202.05441/) (2022)
- [Invariant Causal Prediction for Block MDPs](https://mlsanity.com/paper/2003.06016/) (2020)
- [Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors](https://mlsanity.com/paper/1811.09362/) (2018)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [Desiderata for Representation Learning: A Causal Perspective](https://mlsanity.com/paper/2109.03795/) (2021)
- [Decoupled Multimodal Distilling for Emotion Recognition](https://mlsanity.com/paper/2303.13802/) (2023)
- [Learning Causal Semantic Representation for Out-of-Distribution Prediction](https://mlsanity.com/paper/2011.01681/) (2020)
- [Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion](https://mlsanity.com/paper/1911.07848/) (2019)
- [Active Invariant Causal Prediction: Experiment Selection through Stability](https://mlsanity.com/paper/2006.05690/) (2020)
- [FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Space](https://mlsanity.com/paper/2310.20071/) (2023)
- [Brain encoding models based on multimodal transformers can transfer across language and vision](https://mlsanity.com/paper/2305.12248/) (2023)
- [Invariant Learning via Probability of Sufficient and Necessary Causes](https://mlsanity.com/paper/2309.12559/) (2023)
- [Learning causal representations for robust domain adaptation](https://mlsanity.com/paper/2011.06317/) (2020)
- [On Learning Necessary and Sufficient Causal Graphs](https://mlsanity.com/paper/2301.12389/) (2023)
- [MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks](https://mlsanity.com/paper/2309.14118/) (2023)
- [MuSe-GNN: Learning Unified Gene Representation From Multimodal Biological Graph Data](https://mlsanity.com/paper/2310.02275/) (2023)

## Abstract (arXiv)

Probability of necessity and sufficiency (PNS) measures the likelihood of a feature set being both necessary and sufficient for predicting an outcome. It has proven effective in guiding representation learning for unimodal data, enhancing both predictive performance and model robustness. Despite these benefits, extending PNS to multimodal settings remains unexplored. This extension presents unique challenges, as the conditions for PNS estimation, exogeneity and monotonicity, need to be reconsidered in a multimodal context. We address these challenges by first conceptualizing multimodal representations as comprising modality-invariant and modality-specific components. We then analyze how to compute PNS for each component while ensuring non-trivial PNS estimation. Based on these analyses, we formulate tractable optimization objectives that enable multimodal models to learn high-PNS representations. Experiments demonstrate the effectiveness of our method on both synthetic and real-world data.