mlsanityData to 7 Oct 2026

LLM and GNN are Complementary: Distilling LLM for Multimodal Graph Learning

arXiv 2406.01032 ↗First posted 3 Jun 2024cs.LG cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Recent progress in Graph Neural Networks (GNNs) has greatly enhanced the ability to model complex molecular structures for predicting properties. Nevertheless, molecular data encompasses more than just graph structures, including textual and visual information that GNNs do not handle well. To bridge this gap, we present an innovative framework that utilizes multimodal molecular data to extract insights from Large Language Models (LLMs). We introduce GALLON (Graph Learning from Large Language Model Distillation), a framework that synergizes the capabilities of LLMs and GNNs by distilling multimodal knowledge into a unified Multilayer Perceptron (MLP). This method integrates the rich textual and visual data of molecules with the structural analysis power of GNNs. Extensive experiments reveal that our distilled MLP model notably improves the accuracy and efficiency of molecular property predictions.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

32 of 47 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2017Graph Attention Networkscited by 8,854
  5. 2017Inductive Representation Learning on Large Graphscited by 6,284
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2018How Powerful are Graph Neural Networks?cited by 4,355
  8. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,085
  11. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  12. 2019Strategies for Pre-training Graph Neural Networkscited by 964
20 more
  1. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  2. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  3. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  4. 2022Graph neural networks for materials science and chemistrycited by 181
  5. 2022Translation between Molecules and Natural Languagecited by 173
  6. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  7. 2023Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learningcited by 132
  8. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 129
  9. 2020Distilling Knowledge from Graph Convolutional Networkscited by 128
  10. 2023GraphLLM: Boosting Graph Reasoning Ability of Large Language Modelcited by 105
  11. 2019SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discoverycited by 89
  12. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
  13. 2021Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoodscited by 57
  14. 2023MolFM: A Multimodal Molecular Foundation Modelcited by 50
  15. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  16. 2023Can Large Language Models Empower Molecular Property Prediction?cited by 43
  17. 2023Knowledge Distillation on Graphs: A Surveycited by 38
  18. 2022T2-GNN: Graph Neural Networks for Graphs with Incomplete Features and Structure via Teacher-Student Distillationcited by 32
  19. 2023Which Modality should I use -- Text, Motif, or Image? : Understanding Graphs with Large Language Modelscited by 14
  20. 2023Shape-aware Graph Spectral Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 1.

17 papers.

Sort:

By area:

  1. 2024Unified Parameter-Efficient Unlearning for LLMscited by 24
  2. 2024Decoding Time Series with LLMs: A Multi-Agent Framework for Cross-Domain Annotationcited by 21
  3. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  4. 2025RAG-Enhanced Collaborative LLM Agents for Drug Discoverycited by 18
  5. 2025LanP: Rethinking the Impact of Language Priors in Large Vision-Language Modelscited by 14
  6. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  7. 2024TinyGraph: Joint Feature and Node Condensation for Graph Neural Networkscited by 5
  8. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applicationscited by 5
  9. 2024Demystifying Higher-Order Graph Neural Networkscited by 4
  10. 2025How Far are LLMs from Real Search? A Comprehensive Study on Efficiency, Completeness, and Inherent Capabilitiescited by 4
  11. 2024Stealing Training Graphs from Graph Neural Networkscited by 3
  12. 2025Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Modelscited by 3
5 more
  1. 2024Are You Using Reliable Graph Prompts? Trojan Prompt Attacks on Graph Neural Networkscited by 2
  2. 2025Bridging RDF Knowledge Graphs with Graph Neural Networks for Semantically-Rich Recommender Systemscited by 1
  3. 2025GLOW: Graph-Language Co-Encoding for Agentic Workflow Performance Predictioncited by 1
  4. 2026Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrievalcited by 1
  5. 2025Scalability Matters: Overcoming Challenges in InstructGLM with Similarity-Degree-Based Samplingcited by 0