# RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signals

arXiv: https://arxiv.org/abs/2608.00289 · HTML: https://mlsanity.com/paper/2608.00289/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 41 of its 136 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)
- [CG-HOI: Contact-Guided 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2311.16097/) (2023)
- [Scaling Up Dynamic Human-Scene Interaction Modeling](https://mlsanity.com/paper/2403.08629/) (2024)
- [Full-Body Articulated Human-Object Interaction](https://mlsanity.com/paper/2212.10621/) (2022)
- [Autonomous Character-Scene Interaction Synthesis from Text Instruction](https://mlsanity.com/paper/2410.03187/) (2024)
- [HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environment](https://mlsanity.com/paper/2404.00299/) (2024)
- [RF-Diffusion: Radio Signal Generation via Time-Frequency Diffusion](https://mlsanity.com/paper/2404.09140/) (2024)
- [Human-Object Interaction Detection via Disentangled Transformer](https://mlsanity.com/paper/2204.09290/) (2022)
- [HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Models](https://mlsanity.com/paper/2503.19157/) (2025)
- [InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2509.09555/) (2025)
- [HUMOTO: A 4D Dataset of Mocap Human Object Interactions](https://mlsanity.com/paper/2504.10414/) (2025)
- [NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants](https://mlsanity.com/paper/2301.05223/) (2023)
- [CubeLearn: End-to-end Learning for Human Motion Recognition from Raw mmWave Radar Signals](https://mlsanity.com/paper/2111.03976/) (2021)
- [MiliPoint: A Point Cloud Dataset for mmWave Radar](https://mlsanity.com/paper/2309.13425/) (2023)
- [Visual Content Privacy Protection: A Survey](https://mlsanity.com/paper/2303.16552/) (2023)
- [In-Home Daily-Life Captioning Using Radio Signals](https://mlsanity.com/paper/2008.10966/) (2020)
- [XRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glasses](https://mlsanity.com/paper/2501.19034/) (2025)
- [EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a Wristband](https://mlsanity.com/paper/2401.17409/) (2024)

## Abstract (arXiv)

Recognizing Human-Object Interactions (HOI) is essential for intelligent systems, underpinning applications in virtual and augmented reality, embodied AI, and assistive robotics. However, vision-based HOI methods face challenges in privacy concerns and poor light conditions. In this work, we introduce RF-HOI, the first framework that only uses radio frequency (RF) signals for HOI recognition. A key challenge of RF-HOI is that single-modality RF sensing is insufficient to recognize both actions and the objects being interacted with. RF-HOI addresses this through a novel modality fusion that combines mmWave radar and RFID, enabling simultaneous action recognition and target identification. Another challenge is limited training data across diverse setups, which impairs the generalizability of the recognition model. To overcome this, we develop a simulator that synthesizes multimodal RF data for diverse HOIs at scale, allowing us to fine-tune with only a small amount of real-world data. Experiment results show that RF-HOI outperforms all baselines, approaching vision model performance, and that our diverse synthetic training data can significantly boost our system's performance on real-world scenarios. These results highlight the potential of multimodal RF sensing for robust and privacy-preserving HOI recognition as well as the effectiveness of our RF data synthesis.