# Cross-Modal Hierarchical Fusion for from Multi-Sensor Ground Observation

arXiv: https://arxiv.org/abs/2606.30647 · HTML: https://mlsanity.com/paper/2606.30647/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 31 May 2026.

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks](https://mlsanity.com/paper/1906.05747/) (2019)
- [ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrieval](https://mlsanity.com/paper/2604.17898/) (2026)
- [StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentation](https://mlsanity.com/paper/2408.01343/) (2024)
- [Exploring Efficient Open-Vocabulary Segmentation in the Remote Sensing](https://mlsanity.com/paper/2509.12040/) (2025)
- [HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval](https://mlsanity.com/paper/2604.18037/) (2026)
- [Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrieval](https://mlsanity.com/paper/2604.19386/) (2026)
- [Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies](https://mlsanity.com/paper/2508.10523/) (2025)
- [ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrieval](https://mlsanity.com/paper/2604.20358/) (2026)
- [Exploring the Underwater World Segmentation without Extra Training](https://mlsanity.com/paper/2511.07923/) (2025)
- [CloudSegNet: A Deep Network for Nychthemeron Cloud Image Segmentation](https://mlsanity.com/paper/1904.07979/) (2019)
- [MARIS: Marine Open-Vocabulary Instance Segmentation with Geometric Enhancement and Semantic Alignment](https://mlsanity.com/paper/2510.15398/) (2025)
- [U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation](https://mlsanity.com/paper/2405.15365/) (2024)
- [From Physics to Foundation Models: A Review of AI-Driven Quantitative Remote Sensing Inversion](https://mlsanity.com/paper/2507.09081/) (2025)
- [Multi-view polarimetric scattering cloud tomography and retrieval of droplet size](https://mlsanity.com/paper/2005.11423/) (2020)

## Abstract (arXiv)

Dense volumetric reconstruction of cloud microphysical fields from sparse ground-based instruments remains an open problem, largely because the available measurements are heterogeneous in both modality and spatial coverage. We present AtmoFuseNet, a framework that fuses multi-view sky camera imagery with millimeter-wave cloud radar and ceilometer observations to produce 4D (three spatial dimensions plus time) estimates of cloud state and wind. The method operates in three stages: a cross-modal hierarchical aggregation module that combines image feature pyramids with instrument-derived vertical profiles through layer-wise cross-attention; a conditional variational refinement module that maps the resulting volume to physically consistent microphysical fields under differentiable radar and image forward models; and a correlation-based motion estimator that recovers per-voxel 3D wind vectors from consecutive volumetric reconstructions. On collocated observations from a semi-arid site, AtmoFuseNet reaches 0.026 g m^-3 liquid water content MAE and 1.18 m s^-1 wind speed MAE, improving over existing retrieval baselines. Ablation experiments isolate the contribution of each module.