# ViFiCon: Vision and Wireless Association Via Self-Supervised Contrastive Learning

arXiv: https://arxiv.org/abs/2210.05513 · HTML: https://mlsanity.com/paper/2210.05513/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 68 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [Mobile Sensor Data Anonymization](https://mlsanity.com/paper/1810.11546/) (2018)
- [Contrastive Learning for Image Captioning](https://mlsanity.com/paper/1710.02534/) (2017)
- [CDPAM: Contrastive learning for perceptual audio similarity](https://mlsanity.com/paper/2102.05109/) (2021)
- [Progressive Video Summarization via Multimodal Self-supervised Learning](https://mlsanity.com/paper/2201.02494/) (2022)
- [Self-supervised Feature Learning by Cross-modality and Cross-view Correspondences](https://mlsanity.com/paper/2004.05749/) (2020)
- [AutoFi: Towards Automatic WiFi Human Sensing via Geometric Self-Supervised Learning](https://mlsanity.com/paper/2205.01629/) (2022)
- [Unsupervised Learning for Human Sensing Using Radio Signals](https://mlsanity.com/paper/2207.02370/) (2022)
- [Self-Supervised Radio-Visual Representation Learning for 6G Sensing](https://mlsanity.com/paper/2111.02887/) (2021)

## Built on by (most cited first, 1 of 1)

- [ViFiT: Reconstructing Vision Trajectories from IMU and Wi-Fi Fine Time Measurements](https://mlsanity.com/paper/2310.03140/) (2023)

## Abstract (arXiv)

We introduce ViFiCon, a self-supervised contrastive scheme which learns a cross-modal association between vision and wireless modalities. Specifically, the system uses pedestrian data collected from RGB-D camera footage and WiFi Fine Time Measurements (FTM) from a user's smartphone device. Depth data from RGB-D (vision domain) is inherently linked with an observable pedestrian, but FTM data (wireless domain) is associated only to a smartphone on the network. We represent temporal sequences from both vision and wireless domains by stacking multi-person depth data sequences within an image representation. This simplicity allows both scene-wide processing and fewer vision and wireless features, alleviating privacy and energy associated with transmitting IMU data. To facilitate self-supervised learning, we design a scene-wide synchronization pretext task for our network and then employ the learned representation for the downstream multimodal association task. We show that compared to fully supervised state-of-the-art models, ViFiCon achieves high performance vision-to-wireless association of 92.63% in 25 frames sliding window fashion (2.5s), finding which bounding box corresponds to which smartphone device, without hand-labeled association examples for training data. Extensive experimental results demonstrate ViFiCon applicability in real-world systems when wireless data annotations are scarce.