# EMMA: Extracting Multiple physical parameters from Multimodal Data

arXiv: https://arxiv.org/abs/2605.24047 · HTML: https://mlsanity.com/paper/2605.24047/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 21 May 2026.

## Builds on (most cited first, 21 of 21)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series](https://mlsanity.com/paper/1905.12374/) (2019)
- [Liquid Time-constant Networks](https://mlsanity.com/paper/2006.04439/) (2020)
- [Dynamic mode decomposition with control](https://mlsanity.com/paper/1409.6358/) (2014)
- [SINDy-PI: A Robust Algorithm for Parallel Implicit Sparse Identification of Nonlinear Dynamics](https://mlsanity.com/paper/2004.02322/) (2020)
- [gradSim: Differentiable simulation for system identification and visuomotor control](https://mlsanity.com/paper/2104.02646/) (2021)
- [Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video](https://mlsanity.com/paper/1905.11169/) (2019)
- [RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation](https://mlsanity.com/paper/2205.05678/) (2022)
- [Neural Implicit Representations for Physical Parameter Inference from a Single Video](https://mlsanity.com/paper/2204.14030/) (2022)
- [Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systems](https://mlsanity.com/paper/2410.01376/) (2024)
- [Learning to Identify Physical Parameters from Video Using Differentiable Physics](https://mlsanity.com/paper/2009.08292/) (2020)
- [Digital Twins for Autonomous Driving: A Comprehensive Implementation and Demonstration](https://mlsanity.com/paper/2401.08653/) (2023)
- [A New Era of Mobility: Exploring Digital Twin Applications in Autonomous Vehicular Systems](https://mlsanity.com/paper/2305.16158/) (2023)
- [Recovering implicit physics model under real-world constraints](https://mlsanity.com/paper/2412.02215/) (2024)
- [An approach to improving sound-based vehicle speed estimation](https://mlsanity.com/paper/2204.05082/) (2022)
- [The Unknown Input Observer and its Advantages with Examples](https://mlsanity.com/paper/1504.07300/) (2015)
- [Mel-spectrogram features for acoustic vehicle detection and speed estimation](https://mlsanity.com/paper/2204.04013/) (2022)
- [Model Recovery at the Edge under Resource Constraints for Physical AI](https://mlsanity.com/paper/2512.02283/) (2025)

## Abstract (arXiv)

We introduce EMMA, a physics-informed multimodal framework that recovers all identifiable dynamical parameters of a system directly from raw video, audio, and image-based time-series observations. Unlike prior video-only approaches that struggle with occluded states, hidden actuation inputs, or assumptions about known initial conditions and coordinate frames, EMMA performs joint inference of explicit parameters, implicit dynamical components, and calibration invariants within a unified continuous-time model. EMMA leverages a Liquid Time-Constant (LTC) network to learn latent dynamics from heterogeneous modalities while a physics-constrained loss enforces consistency with the governing differential equations. A unified feature pipeline enables consistent alignment across video trajectories, acoustic signatures, and chart-derived measurements, allowing EMMA to estimate parameters under forced, implicit, and multivariate dynamics without requiring segmentation masks, differentiable rendering, or specialized sensors. Across 100+ scenarios including five standard dynamical benchmarks (75 Delfys videos), real-world rover and quadrotor systems with hidden inputs, and simulation-chart case studies spanning biological and chaotic systems, EMMA delivers robust multi-parameter recovery and significantly outperforms existing single-modality and equation-discovery baselines. Our results establish EMMA as a general, scalable solution for physics-consistent model extraction from opportunistic multimodal data. Code and data are available at: https://github.com/ImpactLabASU/EMMA-CVPR2026