# A Real-Time Gesture-Based Control Framework

arXiv: https://arxiv.org/abs/2504.19460 · HTML: https://mlsanity.com/paper/2504.19460/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 28 Apr 2025.

## Builds on (most cited first, 8 of 8)

- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [Recurrent Human Pose Estimation](https://mlsanity.com/paper/1605.02914/) (2016)
- [Interactive Machine Learning of Musical Gesture](https://mlsanity.com/paper/2011.13487/) (2020)
- [Continuous interaction with a smart speaker via low-dimensional embeddings of dynamic hand pose](https://mlsanity.com/paper/2302.14566/) (2023)
- [Mugeetion: Musical Interface Using Facial Gesture and Emotion](https://mlsanity.com/paper/1809.05502/) (2018)

## Abstract (arXiv)

We introduce a real-time, human-in-the-loop gesture control framework that can dynamically adapt audio and music based on human movement by analyzing live video input. By creating a responsive connection between visual and auditory stimuli, this system enables dancers and performers to not only respond to music but also influence it through their movements. Designed for live performances, interactive installations, and personal use, it offers an immersive experience where users can shape the music in real time. The framework integrates computer vision and machine learning techniques to track and interpret motion, allowing users to manipulate audio elements such as tempo, pitch, effects, and playback sequence. With ongoing training, it achieves user-independent functionality, requiring as few as 50 to 80 samples to label simple gestures. This framework combines gesture training, cue mapping, and audio manipulation to create a dynamic, interactive experience. Gestures are interpreted as input signals, mapped to sound control commands, and used to naturally adjust music elements, showcasing the seamless interplay between human interaction and machine response.