Fusing Cross-Domain Knowledge from Multimodal Data to Solve Problems in the Physical World
Not on the map. Why?
Abstract (arXiv)
The proliferation of artificial intelligence has enabled a diversity of applications that bridge the gap between digital and physical worlds. As physical environments are too complex to model through a single information acquisition approach, it is crucial to fuse multimodal data generated by different sources, such as sensors, devices, systems, and people, to solve a problem in the real world. Unfortunately, it is neither applicable nor sustainable to deploy new resources to collect original data from scratch for every problem. Thus, when data is inadequate in the domain of problem, it is vital to fuse knowledge from multimodal data that is already available in other domains. We call this cross-domain knowledge fusion. Existing research focus on fusing multimodal data in a single domain, supposing the knowledge from different datasets is intrinsically aligned; however, this assumption may not hold in the scenarios of cross-domain knowledge fusion. In this paper, we formally define the cross-domain multimodal data fusion problem, discussing its unique challenges, differences and advantages beyond data fusion in a single domain. We propose a four-layer framework, consisting of Domains, Links, Models and Data layers, answering three key questions:"what to fuse", "why can be fused", and "how to fuse". The Domains Layer selects relevant data from different domains for a given problem. The Links Layer reveals the philosophy of knowledge alignment beyond specific model structures. The Models Layer provides two knowledge fusion paradigms based on the fundamental mechanisms for processing data. The Data Layer turns data of different structures, resolutions, scales and distributions into a consistent representation that can be fed into an AI model. With this framework, we can design solutions that fuse cross-domain multimodal data effectively for solving real-world problems.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 80 references that are in the map. First posted on arXiv 20 May 2025.
Builds on
22 of 80 references.
References · 22
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2021Zero-Shot Text-to-Image Generationcited by 4,288
- 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
- 2020Graph Contrastive Learning with Augmentationscited by 1,208
- 2020Meta-Learning in Neural Networks: A Surveycited by 950
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
- 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
- 2016Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Predictioncited by 333
- 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
10 more
- 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
- 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
- 2019Deep Learning for Spatio-Temporal Data Mining: A Surveycited by 172
- 2013A Survey on Multi-view Learningcited by 152
- 2017A Survey on Network Embeddingcited by 142
- 2023Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Surveycited by 140
- 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 56
- 2022A survey of multimodal deep generative modelscited by 47
- 2021When Do Contrastive Learning Signals Help Spatio-Temporal Graph Forecasting?cited by 36
- 2023Meta-learning approaches for few-shot learning: A survey of recent advancescited by 25
Built on by
Citing papers per year: 2026: 2.
2 papers.