mlsanityData to 14 Oct 2026

Recognizing Co-Speech Gestures in-the-Wild

arXiv 2605.31589 ↗First posted 29 May 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

While humans naturally gesture during speech, only a sparse subset of these co-speech gestures are visually depictive and semantically linked to specific spoken words. In this paper, we introduce a large-scale dataset -- Gesture Recognition in the Wild (GRW), comprising co-speech gestures corresponding to a diverse vocabulary of 155 words. GRW contains 140k manually annotated video clips where the word is spoken, with 17k instances of semantic co-speech gestures including their frame-level temporal boundaries. The video clips are collected 'in the wild' from public-facing discourse, including lectures, talk shows, and interviews, covering a diverse range of speakers and visual conditions. We also introduce video models to: (a) classify gestures as semantic or not; (b) recognize the word corresponding to a co-speech gesture; and (c) temporally localize the gesture. These models are trained and evaluated on the GRW dataset and compared against a range of strong baselines, establishing benchmark results for all three tasks. The dataset, annotations, and trained models are publicly available on the project website: https://www.robots.ox.ac.uk/~vgg/research/grw.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

31 of 49 references.

References · 31

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2025Qwen3-VL Technical Reportcited by 2,793
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  5. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  6. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 583
  7. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 425
  8. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  9. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  10. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 218
  11. 2019Learning Individual Styles of Conversational Gesturecited by 191
  12. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
19 more
  1. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 173
  2. 2019The VIA Annotation Software for Images, Audio and Videocited by 167
  3. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 162
  4. 2023YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpuscited by 75
  5. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  6. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  7. 2025Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Datasetcited by 40
  8. 2019Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networkscited by 30
  9. 2024SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Predictioncited by 22
  10. 2025SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learningcited by 17
  11. 2025SocialGesture: Delving into Multi-person Gesture Understandingcited by 11
  12. 2023Co-Speech Gesture Detection through Multi-Phase Sequence Labelingcited by 5
  13. 2023GestSync: Determining who is speaking without a talking headcited by 4
  14. 2024Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluationcited by 4
  15. 2026SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matchingcited by 4
  16. 2025Understanding Co-speech Gestures in-the-wildcited by 3
  17. 2026SemanticMoments: Training-Free Motion Similarity via Third Moment Featurescited by 3
  18. 2024Leveraging Speech for Gesture Detection in Multimodal Communicationcited by 2
  19. 2025I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialoguecited by 2

Built on by

Not cited yet.