# DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset

arXiv: https://arxiv.org/abs/2211.09769 · HTML: https://mlsanity.com/paper/2211.09769/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 17 Nov 2022.

## Builds on (most cited first, 16 of 16)

- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [DeepMIMO: A Generic Deep Learning Dataset for Millimeter Wave and Massive MIMO Applications](https://mlsanity.com/paper/1902.06435/) (2019)
- [Integrated Sensing and Communications: Towards Dual-functional Wireless Networks for 6G and Beyond](https://mlsanity.com/paper/2108.07165/) (2021)
- [Vision-Aided 6G Wireless Communications: Blockage Prediction and Proactive Handoff](https://mlsanity.com/paper/2102.09527/) (2021)
- [Vision-Position Multi-Modal Beam Prediction Using Real Millimeter Wave Datasets](https://mlsanity.com/paper/2111.07574/) (2021)
- [LiDAR Aided Future Beam Prediction in Real-World Millimeter Wave V2I Communications](https://mlsanity.com/paper/2203.05548/) (2022)
- [ViWi: A Deep Learning Dataset Framework for Vision-Aided Wireless Communications](https://mlsanity.com/paper/1911.06257/) (2019)
- [An Overview of Signal Processing Techniques for Joint Communication and Radar Sensing](https://mlsanity.com/paper/2102.12780/) (2021)
- [A Survey on Fundamental Limits of Integrated Sensing and Communication](https://mlsanity.com/paper/2104.09954/) (2021)
- [Radar Aided 6G Beam Prediction: Deep Learning Algorithms and Real-World Demonstration](https://mlsanity.com/paper/2111.09676/) (2021)
- [Integrated Sensing and Communication for 6G: Ten Key Machine Learning Roles](https://mlsanity.com/paper/2208.02157/) (2022)
- [Multi-Modal Beam Prediction Challenge 2022: Towards Generalization](https://mlsanity.com/paper/2209.07519/) (2022)
- [Computer Vision Aided Blockage Prediction in Real-World Millimeter Wave Deployments](https://mlsanity.com/paper/2203.01907/) (2022)
- [LIDAR and Position-Aided mmWave Beam Selection with Non-local CNNs and Curriculum Training](https://mlsanity.com/paper/2104.14579/) (2021)
- [5G mmWave Cooperative Positioning and Mapping using Multi-Model PHD Filter and Map Fusion](https://mlsanity.com/paper/1908.09806/) (2019)
- [Passive Radar at the Roadside Unit to Configure Millimeter Wave Vehicle-to-Infrastructure Links](https://mlsanity.com/paper/1910.10817/) (2019)

## Built on by (most cited first, 30 of 53)

- [Sionna RT: Differentiable Ray Tracing for Radio Propagation Modeling](https://mlsanity.com/paper/2303.11103/) (2023)
- [Real-Time Digital Twins: Vision and Research Directions for 6G and Beyond](https://mlsanity.com/paper/2301.11283/) (2023)
- [Learning Radio Environments by Differentiable Ray Tracing](https://mlsanity.com/paper/2311.18558/) (2023)
- [A Survey on Integrated Sensing, Communication, and Computation](https://mlsanity.com/paper/2408.08074/) (2024)
- [Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey](https://mlsanity.com/paper/2410.19872/) (2024)
- [Multimodal Transformers for Wireless Communications: A Case Study in Beam Prediction](https://mlsanity.com/paper/2309.11811/) (2023)
- [Multi-Modal Transformer and Reinforcement Learning-based Beam Management](https://mlsanity.com/paper/2410.19859/) (2024)
- [Camera Based mmWave Beam Prediction: Towards Multi-Candidate Real-World Scenarios](https://mlsanity.com/paper/2308.06868/) (2023)
- [Differentiable and Learnable Wireless Simulation with Geometric Transformers](https://mlsanity.com/paper/2406.14995/) (2024)
- [Retrieval Augmented Generation with Multi-Modal LLM Framework for Wireless Environments](https://mlsanity.com/paper/2503.07670/) (2025)
- [BeamLLM: Vision-Empowered mmWave Beam Prediction with Large Language Models](https://mlsanity.com/paper/2503.10432/) (2025)
- [Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework](https://mlsanity.com/paper/2504.05187/) (2025)
- [DeepSense-V2V: A Vehicle-to-Vehicle Multi-Modal Sensing, Localization, and Communications Dataset](https://mlsanity.com/paper/2406.17908/) (2024)
- [M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models](https://mlsanity.com/paper/2506.14532/) (2025)
- [Vehicle Cameras Guide mmWave Beams: Approach and Real-World V2V Demonstration](https://mlsanity.com/paper/2308.10362/) (2023)
- [Joint Sensing and Task-Oriented Communications with Image and Wireless Data Modalities for Dynamic Spectrum Access](https://mlsanity.com/paper/2312.13931/) (2023)
- [CAVIAR: Co-simulation of 6G Communications, 3D Scenarios and AI for Digital Twins](https://mlsanity.com/paper/2401.03310/) (2024)
- [Position Aware 60 GHz mmWave Beamforming for V2V Communications Utilizing Deep Learning](https://mlsanity.com/paper/2402.01259/) (2024)
- [Generative AI Empowered LiDAR Point Cloud Generation with Multimodal Transformer](https://mlsanity.com/paper/2406.18542/) (2024)
- [Beam Selection in ISAC using Contextual Bandit with Multi-modal Transformer and Transfer Learning](https://mlsanity.com/paper/2503.08937/) (2025)
- [Multi-Modality Sensing in mmWave Beamforming for Connected Vehicles Using Deep Learning](https://mlsanity.com/paper/2504.06173/) (2025)
- [Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey](https://mlsanity.com/paper/2601.03181/) (2026)
- [Toward an AI-enabled Connected Industry: AGV Communication and Sensor Measurement Datasets](https://mlsanity.com/paper/2301.03364/) (2022)
- [Enhancing Reliability in Federated mmWave Networks: A Practical and Scalable Solution using Radar-Aided Dynamic Blockage Recognition](https://mlsanity.com/paper/2307.06834/) (2023)
- [Non-contact Multimodal Indoor Human Monitoring Systems: A Survey](https://mlsanity.com/paper/2312.07601/) (2023)
- [Great X: A Unified Multi-Modal Simulator Bridging the Sim2Real Gap for 6G](https://mlsanity.com/paper/2507.08716/) (2025)
- [Large Model Enabled Embodied Intelligence for 6G Integrated Perception, Communication, and Computation Network](https://mlsanity.com/paper/2512.15109/) (2025)
- [Vision-Assisted Digital Twin Creation for mmWave Beam Management](https://mlsanity.com/paper/2401.17781/) (2024)
- [AI-Powered CPS-Enabled Vulnerable-User-Aware Urban Transportation Digital Twin: Methods and Applications](https://mlsanity.com/paper/2501.10396/) (2024)
- [Vision Aided Channel Prediction for Vehicular Communications: A Case Study of Received Power Prediction Using RGB Images](https://mlsanity.com/paper/2501.18618/) (2025)

## Abstract (arXiv)

This article presents the DeepSense 6G dataset, which is a large-scale dataset based on real-world measurements of co-existing multi-modal sensing and communication data. The DeepSense 6G dataset is built to advance deep learning research in a wide range of applications in the intersection of multi-modal sensing, communication, and positioning. This article provides a detailed overview of the DeepSense dataset structure, adopted testbeds, data collection and processing methodology, deployment scenarios, and example applications, with the objective of facilitating the adoption and reproducibility of multi-modal sensing and communication datasets.