# Multimodal Super-Resolution: Discovering hidden physics and its application to fusion plasmas

arXiv: https://arxiv.org/abs/2405.05908 · HTML: https://mlsanity.com/paper/2405.05908/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 92 references that are in the map.
- First posted on arXiv 9 May 2024.

## Builds on (most cited first, 14 of 14)

- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [Context-aware Synthesis for Video Frame Interpolation](https://mlsanity.com/paper/1803.10967/) (2018)
- [Estimating Missing Data in Temporal Data Streams Using Multi-directional Recurrent Neural Networks](https://mlsanity.com/paper/1711.08742/) (2017)
- [Super-Resolution Analysis via Machine Learning: A Survey for Fluid Flows](https://mlsanity.com/paper/2301.10937/) (2023)
- [Deep-Learning-Based Single-Image Height Reconstruction from Very-High-Resolution SAR Intensity Data](https://mlsanity.com/paper/2111.02061/) (2021)
- [From Here to There: Video Inbetweening Using Direct 3D Convolutions](https://mlsanity.com/paper/1905.10240/) (2019)
- [Single-sample image-fusion upsampling of fluorescence lifetime images](https://mlsanity.com/paper/2404.13102/) (2024)

## Built on by (most cited first, 6 of 6)

- [TokaMind: A Multi-Modal Transformer Foundation Model for Tokamak Plasma Dynamics](https://mlsanity.com/paper/2602.15084/) (2026)
- [Improving ideal MHD equilibrium accuracy with physics-informed neural networks](https://mlsanity.com/paper/2507.03119/) (2025)
- [Energy-based models for diagnostic reconstruction and analysis in a laboratory plasma device](https://mlsanity.com/paper/2605.08645/) (2026)
- [Benchmarking Sensor Robustness in Plasma Diagnostic Models: A Systematic Evaluation on TokaMark](https://mlsanity.com/paper/2607.11915/) (2026)
- [Towards Large-Scale Heterogeneous Data Organization for Scientific Foundation Models: A Nuclear Fusion Case Study](https://mlsanity.com/paper/2608.27578/) (2026)
- [IGNITE Tokamak World Model Architecture](https://mlsanity.com/paper/2610.02515/) (2026)

## Abstract (arXiv)

A non-linear system governed by multi-spatial and multi-temporal physics scales cannot be fully understood with a single diagnostic, as each provides only a partial view, leading to information loss. Combining multiple diagnostics may also result in incomplete projections of the system's physics. By identifying hidden inter-correlations between diagnostics, we can leverage mutual support to fill in these gaps, but uncovering such correlations analytically is too complex. We introduce a machine learning methodology to address this issue. Unlike traditional methods, our multimodal approach does not rely on the target diagnostic's direct measurements to generate its super-resolution version. Instead, it uses other diagnostics to produce super-resolution data, capturing detailed structural evolution and responses to perturbations previously unobservable. This not only enhances the resolution of a diagnostic for deeper insights but also reconstructs the target diagnostic, providing a valuable tool to mitigate diagnostic failure. This methodology addresses a key challenge in fusion plasmas: the Edge Localized Mode (ELM), a plasma instability that can cause significant erosion of plasma-facing materials. A method to stabilize ELM is using resonant magnetic perturbation (RMP) to trigger magnetic islands. However, limited spatial and temporal resolution restricts analysis of these islands due to their small size, rapid dynamics, and complex plasma interactions. With super-resolution diagnostics, we can experimentally verify theoretical models of magnetic islands for the first time, providing insights into their role in ELM stabilization. This advancement supports the development of effective ELM suppression strategies for future fusion reactors like ITER and has broader applications, potentially revolutionizing diagnostics in fields such as astronomy, astrophysics, and medical imaging.