# Intelligent Multi-Modal Sensing-Communication Integration: Synesthesia of Machines

arXiv: https://arxiv.org/abs/2306.14143 · HTML: https://mlsanity.com/paper/2306.14143/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2023.

## Built on by (most cited first, 15 of 15)

- [A Survey on Integrated Sensing, Communication, and Computation](https://mlsanity.com/paper/2408.08074/) (2024)
- [Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey](https://mlsanity.com/paper/2410.19872/) (2024)
- [Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework](https://mlsanity.com/paper/2504.05187/) (2025)
- [Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey](https://mlsanity.com/paper/2601.03181/) (2026)
- [Synesthesia of Machines (SoM)-Enhanced ISAC Precoding for Vehicular Networks with Double Dynamics](https://mlsanity.com/paper/2408.13546/) (2024)
- [Multimodal Mixture-of-Experts for ISAC in Low-Altitude Wireless Networks](https://mlsanity.com/paper/2512.01750/) (2025)
- [Doubly-Dynamic ISAC Precoding for Vehicular Networks: A Constrained Deep Reinforcement Learning (CDRL) Approach](https://mlsanity.com/paper/2405.14347/) (2024)
- [Environmental Feature Engineering and Statistical Validation for ML-Based Path Loss Prediction](https://mlsanity.com/paper/2501.08306/) (2025)
- [Synesthesia of Machines (SoM)-Enhanced Sub-THz ISAC Transmission for Air-Ground Network](https://mlsanity.com/paper/2506.12831/) (2025)
- [Synesthesia of Machines (SoM)-Based Task-Driven MIMO System for Image Transmission](https://mlsanity.com/paper/2509.02031/) (2025)
- [Sensing and Understanding the World over Air: A Large Multimodal Model for Mobile Networks](https://mlsanity.com/paper/2511.21707/) (2025)
- [Physic-HM: Restoring Physical Generative Logic in Multimodal Anomaly Detection via Hierarchical Modulation](https://mlsanity.com/paper/2512.21650/) (2025)
- [A Camera-Cooperative ISAC Framework for Multimodal Non-Cooperative UAVs Sensing](https://mlsanity.com/paper/2605.22090/) (2026)
- [Robust Beam Prediction for V2X Networks with Multi-Modal Sensing](https://mlsanity.com/paper/2609.10200/) (2026)
- [Link Both Views: Cross-Image Routing Attention for Dual-Endpoint V2I Beam-Pair Power Prediction](https://mlsanity.com/paper/2610.18195/) (2026)

## Abstract (arXiv)

In the era of sixth-generation (6G) wireless communications, integrated sensing and communications (ISAC) is recognized as a promising solution to upgrade the physical system by endowing wireless communications with sensing capability. Existing ISAC is mainly oriented to static scenarios with radio-frequency (RF) sensors being the primary participants, thus lacking a comprehensive environment feature characterization and facing a severe performance bottleneck in dynamic environments. To date, extensive surveys on ISAC have been conducted but are limited to summarizing RF-based radar sensing. Currently, some research efforts have been devoted to exploring multi-modal sensing-communication integration but still lack a comprehensive review. Therefore, we generalize the concept of ISAC inspired by human synesthesia to establish a unified framework of intelligent multi-modal sensing-communication integration and provide a comprehensive review under such a framework in this paper. The so-termed Synesthesia of Machines (SoM) gives the clearest cognition of such intelligent integration and details its paradigm for the first time. We commence by justifying the necessity of the new paradigm. Subsequently, we offer a definition of SoM and zoom into the detailed paradigm, which is summarized as three operation modes. To facilitate SoM research, we overview the prerequisite of SoM research, i.e., mixed multi-modal (MMM) datasets. Then, we introduce the mapping relationships between multi-modal sensing and communications. Afterward, we cover the technological review on SoM-enhance-based and SoM-concert-based applications. To corroborate the superiority of SoM, we also present simulation results related to dual-function waveform and predictive beamforming design. Finally, we propose some potential directions to inspire future research efforts.