# REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild

arXiv: https://arxiv.org/abs/2403.01229 · HTML: https://mlsanity.com/paper/2403.01229/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 59 references that are in the map.
- First posted on arXiv 2 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [Attention is All You Need in Speech Separation](https://mlsanity.com/paper/2010.13154/) (2020)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection](https://mlsanity.com/paper/1901.01342/) (2019)
- [Pose Flow: Efficient Online Pose Tracking](https://mlsanity.com/paper/1802.00977/) (2018)
- [Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction](https://mlsanity.com/paper/1906.04158/) (2019)
- [rVAD: An Unsupervised Segment-Based Robust Voice Activity Detection Method](https://mlsanity.com/paper/1906.03588/) (2019)
- [PyTorchVideo: A Deep Learning Library for Video Understanding](https://mlsanity.com/paper/2111.09887/) (2021)
- [SALSA: A Novel Dataset for Multimodal Group Behavior Analysis](https://mlsanity.com/paper/1506.06882/) (2015)
- [Cross-modal Supervision for Learning Active Speaker Detection in Video](https://mlsanity.com/paper/1603.08907/) (2016)
- [End-to-end Domain-Adversarial Voice Activity Detection](https://mlsanity.com/paper/1910.10655/) (2019)
- [Defining and Quantifying Conversation Quality in Spontaneous Interactions](https://mlsanity.com/paper/2009.12842/) (2020)
- [Social Processes: Self-Supervised Meta-Learning over Conversational Groups for Forecasting Nonverbal Social Cues](https://mlsanity.com/paper/2107.13576/) (2021)
- [Deep Multimodal Speaker Naming](https://mlsanity.com/paper/1507.04831/) (2015)
- [Perceived Conversation Quality in Spontaneous Interactions](https://mlsanity.com/paper/2207.05791/) (2022)
- [Impact of annotation modality on label quality and model performance in the automatic assessment of laughter in-the-wild](https://mlsanity.com/paper/2211.00794/) (2022)
- [Open Badges: A Low-Cost Toolkit for Measuring Team Communication and Dynamics](https://mlsanity.com/paper/1710.01842/) (2017)
- [No-audio speaking status detection in crowded settings via visual pose-based filtering and wearable acceleration](https://mlsanity.com/paper/2211.00549/) (2022)

## Built on by (most cited first, 1 of 1)

- [COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intention](https://mlsanity.com/paper/2607.28649/) (2026)

## Abstract (arXiv)

Recognizing speaking in humans is a central task towards understanding social interactions. Ideally, speaking would be detected from individual voice recordings, as done previously for meeting scenarios. However, individual voice recordings are hard to obtain in the wild, especially in crowded mingling scenarios due to cost, logistics, and privacy concerns. As an alternative, machine learning models trained on video and wearable sensor data make it possible to recognize speech by detecting its related gestures in an unobtrusive, privacy-preserving way. These models themselves should ideally be trained using labels obtained from the speech signal. However, existing mingling datasets do not contain high quality audio recordings. Instead, speaking status annotations have often been inferred by human annotators from video, without validation of this approach against audio-based ground truth. In this paper we revisit no-audio speaking status estimation by presenting the first publicly available multimodal dataset with high-quality individual speech recordings of 33 subjects in a professional networking event. We present three baselines for no-audio speaking status segmentation: a) from video, b) from body acceleration (chest-worn accelerometer), c) from body pose tracks. In all cases we predict a 20Hz binary speaking status signal extracted from the audio, a time resolution not available in previous datasets. In addition to providing the signals and ground truth necessary to evaluate a wide range of speaking status detection methods, the availability of audio in REWIND makes it suitable for cross-modality studies not feasible with previous mingling datasets. Finally, our flexible data consent setup creates new challenges for multimodal systems under missing modalities.