# PhyMo: A Physical-Field Modality for Multimodal AI4Physics

arXiv: https://arxiv.org/abs/2609.27554 · HTML: https://mlsanity.com/paper/2609.27554/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [xLSTM: Extended Long Short-Term Memory](https://mlsanity.com/paper/2405.04517/) (2024)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [TSMixer: An All-MLP Architecture for Time Series Forecasting](https://mlsanity.com/paper/2303.06053/) (2023)
- [Neural General Circulation Models for Weather and Climate](https://mlsanity.com/paper/2311.07222/) (2023)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [Boreas: A Multi-Season Autonomous Driving Dataset](https://mlsanity.com/paper/2203.10168/) (2022)
- [Self-Supervised Multimodal Learning: A Survey](https://mlsanity.com/paper/2304.01008/) (2023)
- [Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scale](https://mlsanity.com/paper/2603.25040/) (2026)
- [Multimodal Foundation Models for Material Property Prediction and Discovery](https://mlsanity.com/paper/2312.00111/) (2023)
- [SKIPP'D: a SKy Images and Photovoltaic Power Generation Dataset for Short-term Solar Forecasting](https://mlsanity.com/paper/2207.00913/) (2022)
- [A Multimodal PDE Foundation Model for Prediction and Scientific Text Descriptions](https://mlsanity.com/paper/2502.06026/) (2025)
- [Position: Foundation Models Need Digital Twin Representations](https://mlsanity.com/paper/2505.03798/) (2025)
- [RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts](https://mlsanity.com/paper/2508.12291/) (2025)
- [Maven: A Multimodal Foundation Model for Supernova Science](https://mlsanity.com/paper/2408.16829/) (2024)
- [Adapting Vision-Language Models for Neutrino Event Classification in High-Energy Physics](https://mlsanity.com/paper/2509.08461/) (2025)
- [A unified multimodal understanding and generation model for cross-disciplinary scientific research](https://mlsanity.com/paper/2601.01363/) (2026)

## Abstract (arXiv)

Multimodal learning is emerging as a powerful paradigm for AI for Physics (AI4Physics), where predicting physical systems requires the joint interpretation of heterogeneous observations, measurements, and domain knowledge. However, existing approaches typically represent physical quantities and governing equations as generic numerical or textual tokens, overlooking the physical constraints that determine their spatiotemporal interactions. To address this limitation, we introduce the physical-field modality and propose PhyMo, a physics-grounded multimodal framework that organizes heterogeneous measurements through PDE-associated operators. PhyMo follows a three-stage learning procedure: the physical-field encoder is first pretrained through field reconstruction under PDE residual supervision, its representations are subsequently aligned with visual embeddings in a shared latent space, and the fused multimodal representations are finally processed by corresponding downstream prediction heads. Experiments on five datasets spanning diverse physical environments show that PhyMo achieves state-of-the-art performance, compared to the strongest baseline on each dataset, demonstrating the superiority of PhyMo on multimodal representation learning in AI4Physics.