mlsanityData to 7 Oct 2026

Seeking the Sufficiency and Necessity Causal Features in Multimodal Representation Learning

arXiv 2408.16577 ↗First posted 29 Aug 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Probability of necessity and sufficiency (PNS) measures the likelihood of a feature set being both necessary and sufficient for predicting an outcome. It has proven effective in guiding representation learning for unimodal data, enhancing both predictive performance and model robustness. Despite these benefits, extending PNS to multimodal settings remains unexplored. This extension presents unique challenges, as the conditions for PNS estimation, exogeneity and monotonicity, need to be reconsidered in a multimodal context. We address these challenges by first conceptualizing multimodal representations as comprising modality-invariant and modality-specific components. We then analyze how to compute PNS for each component while ensuring non-trivial PNS estimation. Based on these analyses, we formulate tractable optimization objectives that enable multimodal models to learn high-PNS representations. Experiments demonstrate the effectiveness of our method on both synthetic and real-world data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

35 of 48 references.

References · 35

  1. 2019Invariant Risk Minimizationcited by 1,832
  2. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  3. 2018DAGs with NO TEARS: Continuous Optimization for Structure Learningcited by 745
  4. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  5. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 408
  6. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  7. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  8. 2013Causal Discovery with Continuous Additive Noise Modelscited by 252
  9. 2020Invariant Risk Minimization Gamescited by 188
  10. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  11. 2018Learning Factorized Multimodal Representationscited by 171
  12. 2019Causal Discovery from Heterogeneous/Nonstationary Data with Independent Changescited by 166
23 more
  1. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  2. 2019Causal Discovery with Reinforcement Learningcited by 145
  3. 2022Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphscited by 144
  4. 2020Invariant Causal Prediction for Block MDPscited by 104
  5. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  6. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
  7. 2021Desiderata for Representation Learning: A Causal Perspectivecited by 76
  8. 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76
  9. 2020Learning Causal Semantic Representation for Out-of-Distribution Predictioncited by 60
  10. 2019Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusioncited by 45
  11. 2020Active Invariant Causal Prediction: Experiment Selection through Stabilitycited by 37
  12. 2023Brain encoding models based on multimodal transformers can transfer across language and visioncited by 31
  13. 2023FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Spacecited by 31
  14. 2023Invariant Learning via Probability of Sufficient and Necessary Causescited by 22
  15. 2020Learning causal representations for robust domain adaptationcited by 18
  16. 2023On Learning Necessary and Sufficient Causal Graphscited by 14
  17. 2023MultiModN- Multimodal, Multi-Task, Interpretable Modular Networkscited by 14
  18. 2023MuSe-GNN: Learning Unified Gene Representation From Multimodal Biological Graph Datacited by 14
  19. 2023Module-wise Adaptive Distillation for Multimodality Foundation Modelscited by 12
  20. 2018Causal Inference and Mechanism Clustering of A Mixture of Additive Noise Modelscited by 9
  21. 2023Integration-free Training for Spatio-temporal Multimodal Covariate Deep Kernel Point Processescited by 8
  22. 2023Training Transitive and Commutative Multimodal Transformers with LoReTTacited by 4
  23. 2024Unifying Invariance and Spuriousity for Graph Out-of-Distribution via Probability of Necessity and Sufficiencycited by 3

Built on by

Not cited yet.