# Thermodynamics-informed neural networks for physically realistic mixed reality

arXiv: https://arxiv.org/abs/2210.13414 · HTML: https://mlsanity.com/paper/2210.13414/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 22 of 22)

- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [On the Bottleneck of Graph Neural Networks and its Practical Implications](https://mlsanity.com/paper/2006.05205/) (2020)
- [Graph U-Nets](https://mlsanity.com/paper/1905.05178/) (2019)
- [Understanding over-squashing and bottlenecks on graphs via curvature](https://mlsanity.com/paper/2111.14522/) (2021)
- [Physics-informed neural networks (PINNs) for fluid mechanics: A review](https://mlsanity.com/paper/2105.09506/) (2021)
- [Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1708.00588/) (2017)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [Symplectic Recurrent Neural Networks](https://mlsanity.com/paper/1909.13334/) (2019)
- [Hamiltonian Graph Networks with ODE Integrators](https://mlsanity.com/paper/1909.12790/) (2019)
- [Physics-informed neural networks for solving Reynolds-averaged Navierunicodex2013Stokes equations](https://mlsanity.com/paper/2107.10711/) (2021)
- [CROM: Continuous Reduced-Order Modeling of PDEs Using Implicit Neural Representations](https://mlsanity.com/paper/2206.02607/) (2022)
- [Structure-preserving neural networks](https://mlsanity.com/paper/2004.04653/) (2020)
- [Thermodynamics-informed graph neural networks](https://mlsanity.com/paper/2203.01874/) (2022)
- [LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering](https://mlsanity.com/paper/1811.05029/) (2018)
- [Physics-informed deep-learning applications to experimental fluid mechanics](https://mlsanity.com/paper/2203.15402/) (2022)
- [UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality](https://mlsanity.com/paper/2012.03680/) (2020)
- [Augmented Reality Applied to LEGO Construction: AR-based Building Instructions with High Accuracy & Precision and Realistic Object-Hand Occlusions](https://mlsanity.com/paper/1907.12549/) (2019)
- [DeepPhysics: a physics aware deep learning framework for real-time simulation](https://mlsanity.com/paper/2109.09491/) (2021)

## Built on by (most cited first, 9 of 9)

- [Deep Learning in Deterministic Computational Mechanics](https://mlsanity.com/paper/2309.15421/) (2023)
- [P2C2Net: PDE-Preserved Coarse Correction Network for efficient prediction of spatiotemporal dynamics](https://mlsanity.com/paper/2411.00040/) (2024)
- [Physics-informed Deep Learning for Muscle Force Prediction with Unlabeled sEMG Signals](https://mlsanity.com/paper/2412.04213/) (2024)
- [Conservation-informed Graph Learning for Spatiotemporal Dynamics Prediction](https://mlsanity.com/paper/2412.20962/) (2024)
- [On the under-reaching phenomenon in message-passing neural PDE solvers: revisiting the CFL condition](https://mlsanity.com/paper/2507.08861/) (2025)
- [A hybrid numerical methodology coupling Reduced Order Modeling and Graph Neural Networks for non-parametric geometries: applications to structural dynamics problems](https://mlsanity.com/paper/2406.02615/) (2024)
- [MultiPDENet: PDE-embedded Learning with Multi-time-stepping for Accelerated Flow Simulation](https://mlsanity.com/paper/2501.15987/) (2025)
- [Thermodynamics-informed graph neural networks for real-time simulation of digital human twins](https://mlsanity.com/paper/2412.12034/) (2024)
- [Learnable-Differentiable Finite Volume Solver for Accelerated Simulation of Flows](https://mlsanity.com/paper/2507.01975/) (2025)

## Abstract (arXiv)

The imminent impact of immersive technologies in society urges for active research in real-time and interactive physics simulation for virtual worlds to be realistic. In this context, realistic means to be compliant to the laws of physics. In this paper we present a method for computing the dynamic response of (possibly non-linear and dissipative) deformable objects induced by real-time user interactions in mixed reality using deep learning. The graph-based architecture of the method ensures the thermodynamic consistency of the predictions, whereas the visualization pipeline allows a natural and realistic user experience. Two examples of virtual solids interacting with virtual or physical solids in mixed reality scenarios are provided to prove the performance of the method.