# SocioGesture: Real-Time and Adaptive Social Gesture Perception for Human-Robot Interaction

arXiv: https://arxiv.org/abs/2609.04545 · HTML: https://mlsanity.com/paper/2609.04545/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [PYSKL: Towards Good Practices for Skeleton Action Recognition](https://mlsanity.com/paper/2205.09443/) (2022)
- [Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks](https://mlsanity.com/paper/2105.12374/) (2021)
- [Active Learning in Robotics: A Review of Control Principles](https://mlsanity.com/paper/2106.13697/) (2021)
- [RTMW: Real-Time Multi-Person 2D and 3D Whole-body Pose Estimation](https://mlsanity.com/paper/2407.08634/) (2024)
- [Diffusion Models on the Edge: Challenges, Optimizations, and Applications](https://mlsanity.com/paper/2504.15298/) (2025)
- [Body-Hand Modality Expertized Networks with Cross-attention for Fine-grained Skeleton Action Recognition](https://mlsanity.com/paper/2503.14960/) (2025)
- [Hand and Arm Gesture-based Human-Robot Interaction: A Review](https://mlsanity.com/paper/2209.08229/) (2022)
- [CAFL-L: Constraint-Aware Federated Learning with Lagrangian Dual Optimization for On-Device Language Models](https://mlsanity.com/paper/2510.03298/) (2025)

## Abstract (arXiv)

Robots interacting with people must recognize not only explicit commands, but also social cues such as invitations, refusals, and unavailability. In real deployments, these cues must be inferred from noisy onboard perception under partial occlusion, changing viewpoints, and strict latency constraints. We present SocioGesture, a real-time adaptive social gesture perception system for human-robot interaction (HRI). SocioGesture uses a compact confidence-aware body-hand skeleton representation and a lightweight dual-stream model that fuses body motion with hand articulation for low-latency onboard recognition. To improve deployment robustness, we train the model with occlusion-aware skeleton corruption, exposing it to missing hands, occluded arms, and temporally unstable keypoints without increasing the inference cost. On a social gesture dataset collected in mixed indoor-outdoor HRI scenarios, SocioGesture achieves strong held-out-subject recognition, substantially improves robustness under structured joint occlusion, and runs in real time on a robot-mounted edge device. During deployment, uncertain interaction segments are saved for offline labeling and adaptation, enabling SocioGesture to expand its gesture vocabulary while preserving performance in the original classes. These results demonstrate a practical path toward robust, efficient, and adaptive social perception for interactive robots.