# Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2407.18999 · HTML: https://mlsanity.com/paper/2407.18999/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 52 of its 81 references that are in the map.
- First posted on arXiv 26 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 52)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Diffusion Models already have a Semantic Latent Space](https://mlsanity.com/paper/2210.10960/) (2022)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1808.00948/) (2018)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Interpreting CLIP's Image Representation via Text-Based Decomposition](https://mlsanity.com/paper/2310.05916/) (2023)
- [TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document](https://mlsanity.com/paper/2403.04473/) (2024)
- [Towards Unsupervised Deep Graph Structure Learning](https://mlsanity.com/paper/2201.06367/) (2022)
- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations](https://mlsanity.com/paper/1705.08841/) (2017)
- [Progressive Face Super-Resolution via Attention to Facial Landmark](https://mlsanity.com/paper/1908.08239/) (2019)
- [Disentangled Representation Learning in Cardiac Image Analysis](https://mlsanity.com/paper/1903.09467/) (2019)
- [Weakly Supervised Disentangled Generative Causal Representation Learning](https://mlsanity.com/paper/2010.02637/) (2020)
- [Image-to-image Translation via Hierarchical Style Disentanglement](https://mlsanity.com/paper/2103.01456/) (2021)
- [Image-to-image translation for cross-domain disentanglement](https://mlsanity.com/paper/1805.09730/) (2018)

## Built on by (most cited first, 10 of 10)

- [Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement](https://mlsanity.com/paper/2506.01663/) (2025)
- [Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learning](https://mlsanity.com/paper/2503.08751/) (2025)
- [UMMAN: Unsupervised Multi-graph Merge Adversarial Network for Disease Prediction Based on Intestinal Flora](https://mlsanity.com/paper/2407.21714/) (2024)
- [Interpretable Single-View 3D Gaussian Splatting using Unsupervised Hierarchical Disentangled Representation Learning](https://mlsanity.com/paper/2504.04190/) (2025)
- [LiveGraph: Active-Structure Neural Re-ranking for Exercise Recommendation](https://mlsanity.com/paper/2602.17036/) (2026)
- [The 1st International Workshop on Disentangled Representation Learning for Controllable Generation (DRL4Real): Methods and Results](https://mlsanity.com/paper/2509.10463/) (2025)
- [FusionGen: Feature Fusion-Based Few-Shot EEG Data Generation](https://mlsanity.com/paper/2510.10604/) (2025)
- [Disentangle-then-Refine: LLM-Guided Decoupling and Structure-Aware Refinement for Graph Contrastive Learning](https://mlsanity.com/paper/2604.14746/) (2026)
- [Controlla: Learning Controllability via Graph-Constrained Latent Geometry](https://mlsanity.com/paper/2605.16603/) (2026)
- [Coordinated Disentanglement with Iterative Mode Discovery Under Hidden Correlations](https://mlsanity.com/paper/2607.17264/) (2026)

## Abstract (arXiv)

Disentangled representation learning (DRL) aims to identify and decompose underlying factors behind observations, thus facilitating data perception and generation. However, current DRL approaches often rely on the unrealistic assumption that semantic factors are statistically independent. In reality, these factors may exhibit correlations, which off-the-shelf solutions have yet to properly address. To tackle this challenge, we introduce a bidirectional weighted graph-based framework, to learn factorized attributes and their interrelations within complex data. Specifically, we propose a β-VAE based module to extract factors as the initial nodes of the graph, and leverage the multimodal large language model (MLLM) to discover and rank latent correlations, thereby updating the weighted edges. By integrating these complementary modules, our model successfully achieves fine-grained, practical and unsupervised disentanglement. Experiments demonstrate our method's superior performance in disentanglement and reconstruction. Furthermore, the model inherits enhanced interpretability and generalizability from MLLMs.