# Joint Sensing and Task-Oriented Communications with Image and Wireless Data Modalities for Dynamic Spectrum Access arXiv: https://arxiv.org/abs/2312.13931 ยท HTML: https://mlsanity.com/paper/2312.13931/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 27 of its 33 references that are in the map. - First posted on arXiv 21 Dec 2023. ## Builds on (most cited first, 27 of 27) - [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020) - [An Introduction to Deep Learning for the Physical Layer](https://mlsanity.com/paper/1702.00832/) (2017) - [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022) - [6G Networks: Beyond Shannon Towards Semantic and Goal-Oriented Communications](https://mlsanity.com/paper/2011.14844/) (2020) - [Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach](https://mlsanity.com/paper/2102.04170/) (2021) - [Integrated Sensing and Communications: Towards Dual-functional Wireless Networks for 6G and Beyond](https://mlsanity.com/paper/2108.07165/) (2021) - [Semantic Communications: Principles and Challenges](https://mlsanity.com/paper/2201.01389/) (2021) - [Task-Oriented Multi-User Semantic Communications](https://mlsanity.com/paper/2112.10255/) (2021) - [Less Data, More Knowledge: Building Next Generation Semantic Communication Networks](https://mlsanity.com/paper/2211.14343/) (2022) - [DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset](https://mlsanity.com/paper/2211.09769/) (2022) - [Task-Oriented Multi-User Semantic Communications for VQA Task](https://mlsanity.com/paper/2108.07357/) (2021) - [Deep Learning for Wireless Communications](https://mlsanity.com/paper/2005.06068/) (2020) - [Deep Learning-Enabled Semantic Communication Systems with Task-Unaware Transmitter and Dynamic Data](https://mlsanity.com/paper/2205.00271/) (2022) - [Task-Oriented Image Transmission for Scene Classification in Unmanned Aerial Systems](https://mlsanity.com/paper/2112.10948/) (2021) - [An Overview of Signal Processing Techniques for Joint Communication and Radar Sensing](https://mlsanity.com/paper/2102.12780/) (2021) - [Integrating Sensing and Communications for Ubiquitous IoT: Applications, Trends and Challenges](https://mlsanity.com/paper/2104.11457/) (2021) - [Task-Oriented Communications for NextG: End-to-End Deep Learning and AI Security Aspects](https://mlsanity.com/paper/2212.09668/) (2022) - [Learning-based Predictive Beamforming for Integrated Sensing and Communication in Vehicular Networks](https://mlsanity.com/paper/2108.11540/) (2021) - [Integrated Sensing and Communication for 6G: Ten Key Machine Learning Roles](https://mlsanity.com/paper/2208.02157/) (2022) - [Is Semantic Communications Secure? A Tale of Multi-Domain Adversarial Attacks](https://mlsanity.com/paper/2212.10438/) (2022) - [Computer Vision Aided Blockage Prediction in Real-World Millimeter Wave Deployments](https://mlsanity.com/paper/2203.01907/) (2022) - [Age of Information in Deep Learning-Driven Task-Oriented Communications](https://mlsanity.com/paper/2301.04298/) (2023) - [Joint Sensing and Semantic Communications with Multi-Task Deep Learning](https://mlsanity.com/paper/2311.05017/) (2023) - [End-to-End Learning for Integrated Sensing and Communication](https://mlsanity.com/paper/2111.02106/) (2021) - [Timeliness of Information for Computation-intensive Status Updates in Task-oriented Communications](https://mlsanity.com/paper/2209.13259/) (2022) - [Integrated Sensing and Communications for V2I Networks: Dynamic Predictive Beamforming for Extended Vehicle Targets](https://mlsanity.com/paper/2111.10152/) (2021) - [Autoencoder-based Joint Communication and Sensing of Multiple Targets](https://mlsanity.com/paper/2301.09439/) (2023) ## Built on by (most cited first, 3 of 3) - [Low-Latency Task-Oriented Communications with Multi-Round, Multi-Task Deep Learning](https://mlsanity.com/paper/2411.10385/) (2024) - [When Semantic Communication Meets Queueing: Cross-Layer Latency and Task Fidelity Optimization](https://mlsanity.com/paper/2605.05514/) (2026) - [Resolving Multi-Target Association in OFDM-based ISAC via Vision-aided Multi-Modal Learning](https://mlsanity.com/paper/2606.22195/) (2026) ## Abstract (arXiv) This paper introduces a deep learning approach to dynamic spectrum access, leveraging the synergy of multi-modal image and spectrum data for the identification of potential transmitters. We consider an edge device equipped with a camera that is taking images of potential objects such as vehicles that may harbor transmitters. Recognizing the computational constraints and trust issues associated with on-device computation, we propose a collaborative system wherein the edge device communicates selectively processed information to a trusted receiver acting as a fusion center, where a decision is made to identify whether a potential transmitter is present, or not. To achieve this, we employ task-oriented communications, utilizing an encoder at the transmitter for joint source coding, channel coding, and modulation. This architecture efficiently transmits essential information of reduced dimension for object classification. Simultaneously, the transmitted signals may reflect off objects and return to the transmitter, allowing for the collection of target sensing data. Then the collected sensing data undergoes a second round of encoding at the transmitter, with the reduced-dimensional information communicated back to the fusion center through task-oriented communications. On the receiver side, a decoder performs the task of identifying a transmitter by fusing data received through joint sensing and task-oriented communications. The two encoders at the transmitter and the decoder at the receiver are jointly trained, enabling a seamless integration of image classification and wireless signal detection. Using AWGN and Rayleigh channel models, we demonstrate the effectiveness of the proposed approach, showcasing high accuracy in transmitter identification across diverse channel conditions while sustaining low latency in decision making.