mlsanityData to 14 Oct 2026

FusDreamer: Label-efficient Remote Sensing World Model for Multimodal Data Classification

arXiv 2503.13814 ↗First posted 18 Mar 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

World models significantly enhance hierarchical understanding, improving data integration and learning efficiency. To explore the potential of the world model in the remote sensing (RS) field, this paper proposes a label-efficient remote sensing world model for multimodal data fusion (FusDreamer). The FusDreamer uses the world model as a unified representation container to abstract common and high-level knowledge, promoting interactions across different types of data, i.e., hyperspectral (HSI), light detection and ranging (LiDAR), and text data. Initially, a new latent diffusion fusion and multimodal generation paradigm (LaMG) is utilized for its exceptional information integration and detail retention capabilities. Subsequently, an open-world knowledge-guided consistency projection (OK-CP) module incorporates prompt representations for visually described objects and aligns language-visual features through contrastive learning. In this way, the domain gap can be bridged by fine-tuning the pre-trained world models with limited samples. Finally, an end-to-end multitask combinatorial optimization (MuCO) strategy can capture slight feature bias and constrain the diffusion process in a collaboratively learnable direction. Experiments conducted on four typical datasets indicate the effectiveness and advantages of the proposed FusDreamer. The corresponding code will be released at https://github.com/Cimy-wang/FusDreamer.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 66 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

34 of 66 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  8. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  9. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  10. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  11. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  12. 2020A Survey on Visual Transformercited by 654
22 more
  1. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  2. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  3. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  4. 2022CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusioncited by 223
  5. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 191
  6. 2023DDFM: Denoising Diffusion Model for Multi-Modality Image Fusioncited by 153
  7. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 143
  8. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  9. 2019Deep Learning for Hyperspectral Image Classification: An Overviewcited by 123
  10. 2023Equivariant Multi-Modality Image Fusioncited by 99
  11. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  12. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Modelcited by 45
  13. 2024Image Fusion via Vision-Language Modelcited by 40
  14. 2022ScalableViT: Rethinking the Context-oriented Generalization of Vision Transformercited by 36
  15. 2022Language-aware Domain Generalization Network for Cross-Scene Hyperspectral Image Classificationcited by 20
  16. 2023Nearest Neighbor-Based Contrastive Learning for Hyperspectral and LiDAR Data Classificationcited by 12
  17. 2023DCN-T: Dual Context Network with Transformer for Hyperspectral Image Classificationcited by 12
  18. 2022CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioningcited by 11
  19. 2022A3CLNN: Spatial, Spectral and Multiscale Attention ConvLSTM Neural Network for Multisource Remote Sensing Data Classificationcited by 9
  20. 2022Exploring Cross-Domain Pretrained Model for Hyperspectral Image Classificationcited by 5
  21. 2024Multimodal Informative ViT: Information Aggregation and Distribution for Hyperspectral and LiDAR Classificationcited by 5
  22. 2021ASPCNet: A Deep Adaptive Spatial Pattern Capsule Network for Hyperspectral Image Classificationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Balanced Diffusion-Guided Fusion for Multimodal Remote Sensing Classification
  2. 2025Frequency-Aware Vision-Language Multimodality Generalization Network for Remote Sensing Image Classification
  3. 2024From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing
  4. 2026FusionRS: A Large-Scale RGB-Infrared-Style Remote Sensing Dataset for Cross-Modal Vision-Language Learning
  5. 2023FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients
  6. 2026Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?
  7. 2026Omni-RS: Unifying Multi-Task Understanding and Generation in Remote Sensing
  8. 2025A Diff-Attention Aware State Space Fusion Model for Remote Sensing Classification