mlsanityData to 7 Oct 2026

Multimodal learning with graphs

arXiv 2209.03299 ↗First posted 7 Sep 2022cs.LG cs.AIcited by 54

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence for graphs has achieved remarkable success in modeling complex systems, ranging from dynamic networks in biology to interacting particle systems in physics. However, the increasingly heterogeneous graph datasets call for multimodal methods that can combine different inductive biases: the set of assumptions that algorithms use to make predictions for inputs they have not encountered during training. Learning on multimodal datasets presents fundamental challenges because the inductive biases can vary by data modality and graphs might not be explicitly given in the input. To address these challenges, multimodal graph AI methods combine different modalities while leveraging cross-modal dependencies using graphs. Diverse datasets are combined using graphs and fed into sophisticated multimodal architectures, specified as image-intensive, knowledge-grounded and language-intensive models. Using this categorization, we introduce a blueprint for multimodal graph learning, use it to study existing methods and provide guidelines to design new models.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 99 of its 149 references that are in the map. First posted on arXiv 7 Sep 2022.

Builds on

99 of 149 references.

References · 99

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2017Graph Attention Networkscited by 8,854
  6. 2017Inductive Representation Learning on Large Graphscited by 6,284
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  8. 2018How Powerful are Graph Neural Networks?cited by 4,355
  9. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  10. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,085
  11. 2019Simplifying Graph Convolutional Networkscited by 1,470
  12. 2016Variational Graph Auto-Encoderscited by 1,373
38 more
  1. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  2. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  3. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  4. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  5. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  6. 2019Relational Knowledge Distillationcited by 873
  7. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  8. 2018Link Prediction Based on Graph Neural Networkscited by 829
  9. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  10. 2020Directional Message Passing for Molecular Graphscited by 556
  11. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  12. 2018Hypergraph Neural Networkscited by 482
  13. 2020Principal Neighbourhood Aggregation for Graph Netscited by 477
  14. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  15. 2022Multimodal Learning with Transformers: A Surveycited by 399
  16. 2018Graph Convolutional Networks for Text Classificationcited by 345
  17. 2017Residual Gated Graph ConvNetscited by 341
  18. 2019Scene Text Visual Question Answeringcited by 327
  19. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  20. 2015Deeply-Recursive Convolutional Network for Image Super-Resolutioncited by 317
  21. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  22. 2018Graph networks as learnable physics engines for inference and controlcited by 242
  23. 2015Order Matters: Sequence to sequence for setscited by 236
  24. 2021Weisfeiler and Lehman Go Cellular: CW Networkscited by 218
  25. 2017CayleyNets: Graph Convolutional Neural Networks with Complex Rational Spectral Filterscited by 203
  26. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  27. 2020Directional Graph Networkscited by 132
  28. 2018Graph-Based Global Reasoning Networkscited by 120
  29. 2019Neural Execution of Graph Algorithmscited by 118
  30. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  31. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  32. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  33. 2019Edge-labeling Graph Neural Network for Few-shot Learningcited by 114
  34. 2020Graph Convolutional Networks for Hyperspectral Image Classificationcited by 113
  35. 2018Graphite: Iterative Generative Modeling of Graphscited by 112
  36. 2021Graph-Guided Network for Irregularly Sampled Multivariate Time Seriescited by 110
  37. 2021Combinatorial Optimization with Physics-Inspired Graph Neural Networkscited by 107
  38. 2020Cross-Scale Internal Graph Neural Network for Image Super-Resolutioncited by 94

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 20; 2025: 14; 2026: 10.

54 papers.

Sort:

By area:

  1. 2025UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphscited by 47
  2. 2024Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learningcited by 28
  3. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  4. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  5. 2024Learning on Multimodal Graphs: A Surveycited by 22
  6. 2024Networking Systems for Video Anomaly Detection: A Tutorial and Surveycited by 21
  7. 2023Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Reviewcited by 20
  8. 2024Graph Learning under Distribution Shifts: A Comprehensive Survey on Domain Adaptation, Out-of-distribution, and Continual Learningcited by 17
  9. 2023MuSe-GNN: Learning Unified Gene Representation From Multimodal Biological Graph Datacited by 14
  10. 2025Graph World Modelcited by 14
  11. 2025Cross-Contrastive Clustering for Multimodal Attributed Graphs with Dual Graph Filteringcited by 14
  12. 2024Graph Neural Networks in Histopathology: Emerging Trends and Future Directionscited by 13
38 more
  1. 2024Towards Modality Generalization: A Benchmark and Prospective Analysiscited by 10
  2. 2024MM-GTUNets: Unified Multi-Modal Graph Deep Learning for Brain Disorders Predictioncited by 9
  3. 2023Exploring Causal Learning through Graph Neural Networks: An In-depth Reviewcited by 8
  4. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  5. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  6. 2025A Comprehensive Data-centric Overview of Federated Graph Learningcited by 7
  7. 2023Building Flexible, Scalable, and Machine Learning-ready Multimodal Oncology Datasetscited by 6
  8. 2023BioBLP: A Modular Framework for Learning on Multimodal Biomedical Knowledge Graphscited by 5
  9. 2023Advancing Drug Discovery with Enhanced Chemical Understanding via Asymmetric Contrastive Multimodal Learningcited by 5
  10. 2023Graph AI in Medicinecited by 3
  11. 2024Recent Advances, Applications, and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposiumcited by 3
  12. 2023Geometric Deep Learning for Structure-Based Drug Design: A Surveycited by 2
  13. 2024Generating Realistic X-ray Scattering Images Using Stable Diffusion and Human-in-the-loop Annotationscited by 2
  14. 2025Towards deployment-centric multimodal AI beyond vision and languagecited by 2
  15. 2025The Tsetlin Machine Goes Deep: Logical Learning and Reasoning With Graphscited by 2
  16. 2025Interpretable Generative and Discriminative Learning for Multimodal and Incomplete Clinical Datacited by 2
  17. 2023Knowledge Graphs are not Created Equal: Exploring the Properties and Structure of Real KGscited by 1
  18. 2024Uncertainty-Aware Hardware Trojan Detection Using Multimodal Deep Learningcited by 1
  19. 2024Stitching Gaps: Fusing Situated Perceptual Knowledge with Vision Transformers for High-Level Image Classificationcited by 1
  20. 2024Explaining Hypergraph Neural Networks: From Local Explanations to Global Conceptscited by 1
  21. 2024Revealing the Evolution of Order in Materials Microstructures Using Multi-Modal Computer Visioncited by 1
  22. 2025Rethinking industrial artificial intelligence: a unified foundation frameworkcited by 1
  23. 2025Grounding the Ungrounded: A Spectral-Graph Framework for Quantifying Hallucinations in Multimodal LLMscited by 1
  24. 2026Modality as Heterogeneity: Node Splitting and Graph Rewiring for Multimodal Graph Learningcited by 1
  25. 2026A Doubly Robust Machine Learning Approach for Disentangling Treatment Effect Heterogeneity with Functional Outcomescited by 1
  26. 2026Attention-based graph neural networks: a surveycited by 1
  27. 2026Towards Robust Federated Multimodal Graph Learning under Modality Heterogeneitycited by 1
  28. 2026Context-aware Modality-Topology Co-Alignment for Multimodal Attributed Graphscited by 1
  29. 2024Mixture of multilayer stochastic block models for multiview clusteringcited by 0
  30. 2024Explaining Graph Neural Networks for Node Similarity on Graphscited by 0
  31. 2024Probability Passing for Graph Neural Networks: Graph Structure and Representations Joint Learningcited by 0
  32. 2024Heterogeneous graph attention network improves cancer multiomics integrationcited by 0
  33. 2024Boolean Product Graph Neural Networkscited by 0
  34. 2025Engineering Artificial Intelligence: Framework, Challenges, and Future Directioncited by 0
  35. 2025Towards Multimodal Graph Large Language Modelcited by 0
  36. 2025Knowledge Graph Sparsification for GNN-based Rare Disease Diagnosiscited by 0
  37. 2025Learning Multimodal Embeddings for Traffic Accident Prediction and Causal Estimationcited by 0
  38. 2026Bayesian Integration of Nonlinear Incomplete Clinical Datacited by 0

50 most cited of 54 All 54 (JSON)