# Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discovery

arXiv: https://arxiv.org/abs/2202.03242 · HTML: https://mlsanity.com/paper/2202.03242/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1611.02648/) (2016)
- [Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering](https://mlsanity.com/paper/1611.05148/) (2016)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)
- [Variational Autoencoder for Deep Learning of Images, Labels and Captions](https://mlsanity.com/paper/1609.08976/) (2016)
- [Joint Multimodal Learning with Deep Generative Models](https://mlsanity.com/paper/1611.01891/) (2016)
- [Generalized Multimodal ELBO](https://mlsanity.com/paper/2105.02470/) (2021)
- [Generative Models of Visually Grounded Imagination](https://mlsanity.com/paper/1705.10762/) (2017)
- [An Ensemble of Simple Convolutional Neural Network Models for MNIST Digit Recognition](https://mlsanity.com/paper/2008.10400/) (2020)

## Built on by (most cited first, 6 of 6)

- [Recent Advances and Applications of Machine Learning in Experimental Solid Mechanics: A Review](https://mlsanity.com/paper/2303.07647/) (2023)
- [Physics-Informed Latent Diffusion for Multimodal Brain MRI Synthesis](https://mlsanity.com/paper/2409.13532/) (2024)
- [G2Phinet: Relating Genotype and Biomechanical Phenotype of Tissues with Deep Learning](https://mlsanity.com/paper/2208.09889/) (2022)
- [Physics-informed Variational Autoencoders for Improved Robustness to Environmental Factors of Variation](https://mlsanity.com/paper/2210.10418/) (2022)
- [Causal disentanglement of multimodal data](https://mlsanity.com/paper/2310.18471/) (2023)
- [Physically Interpretable Representation Learning with Gaussian Mixture Variational AutoEncoder (GM-VAE)](https://mlsanity.com/paper/2511.21883/) (2025)

## Abstract (arXiv)

We introduce physics-informed multimodal autoencoders (PIMA) - a variational inference framework for discovering shared information in multimodal scientific datasets representative of high-throughput testing. Individual modalities are embedded into a shared latent space and fused through a product of experts formulation, enabling a Gaussian mixture prior to identify shared features. Sampling from clusters allows cross-modal generative modeling, with a mixture of expert decoder imposing inductive biases encoding prior scientific knowledge and imparting structured disentanglement of the latent space. This approach enables discovery of fingerprints which may be detected in high-dimensional heterogeneous datasets, avoiding traditional bottlenecks related to high-fidelity measurement and characterization. Motivated by accelerated co-design and optimization of materials manufacturing processes, a dataset of lattice metamaterials from metal additive manufacturing demonstrates accurate cross modal inference between images of mesoscale topology and mechanical stress-strain response.