# Streamlining Multimodal Data Fusion in Wireless Communication and Sensor Networks

arXiv: https://arxiv.org/abs/2302.12636 · HTML: https://mlsanity.com/paper/2302.12636/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 49 references that are in the map.
- First posted on arXiv 24 Feb 2023.

## Builds on (most cited first, 19 of 19)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Joint Multimodal Learning with Deep Generative Models](https://mlsanity.com/paper/1611.01891/) (2016)
- [Generalized Multimodal ELBO](https://mlsanity.com/paper/2105.02470/) (2021)
- [Exploring Contrastive Learning in Human Activity Recognition for Healthcare](https://mlsanity.com/paper/2011.11542/) (2020)
- [Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2012.05333/) (2020)
- [Vector Quantized Models for Planning](https://mlsanity.com/paper/2106.04615/) (2021)
- [Overview of Deep Learning-based CSI Feedback in Massive MIMO Systems](https://mlsanity.com/paper/2206.14383/) (2022)
- [EfficientFi: Towards Large-Scale Lightweight WiFi Sensing via CSI Compression](https://mlsanity.com/paper/2204.04138/) (2022)
- [Multi-resolution CSI Feedback with deep learning in Massive MIMO System](https://mlsanity.com/paper/1910.14322/) (2019)
- [Gimme Signals: Discriminative signal encoding for multimodal activity recognition](https://mlsanity.com/paper/2003.06156/) (2020)
- [Lightweight Convolutional Neural Networks for CSI Feedback in Massive MIMO](https://mlsanity.com/paper/2005.00438/) (2020)
- [Multimodal sensor fusion in the latent representation space](https://mlsanity.com/paper/2208.02183/) (2022)
- [PRVNet: A Novel Partially-Regularized Variational Autoencoders for Massive MIMO CSI Feedback](https://mlsanity.com/paper/2011.04178/) (2020)
- [Self-Supervised WiFi-Based Activity Recognition](https://mlsanity.com/paper/2104.09072/) (2021)
- [Self-Supervised Multimodal Fusion Transformer for Passive Activity Recognition](https://mlsanity.com/paper/2209.03765/) (2022)
- [Learning Representations for CSI Adaptive Quantization and Feedback](https://mlsanity.com/paper/2207.06924/) (2022)

## Built on by (most cited first, 2 of 2)

- [Modeling Spatio-temporal Dynamical Systems with Neural Discrete Learning and Levels-of-Experts](https://mlsanity.com/paper/2402.05970/) (2024)
- [Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey](https://mlsanity.com/paper/2601.03181/) (2026)

## Abstract (arXiv)

This paper presents a novel approach for multimodal data fusion based on the Vector-Quantized Variational Autoencoder (VQVAE) architecture. The proposed method is simple yet effective in achieving excellent reconstruction performance on paired MNIST-SVHN data and WiFi spectrogram data. Additionally, the multimodal VQVAE model is extended to the 5G communication scenario, where an end-to-end Channel State Information (CSI) feedback system is implemented to compress data transmitted between the base-station (eNodeB) and User Equipment (UE), without significant loss of performance. The proposed model learns a discriminative compressed feature space for various types of input data (CSI, spectrograms, natural images, etc), making it a suitable solution for applications with limited computational resources.