# A Benchmark for Spatially Grounded Gesture Generation

arXiv: https://arxiv.org/abs/2610.03105 · HTML: https://mlsanity.com/paper/2610.03105/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Builds on (most cited first, 21 of 21)

- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [OmniControl: Control Any Joint at Any Time for Human Motion Generation](https://mlsanity.com/paper/2310.08580/) (2023)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents](https://mlsanity.com/paper/2303.14613/) (2023)
- [EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling](https://mlsanity.com/paper/2401.00374/) (2023)
- [Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance](https://mlsanity.com/paper/2403.18036/) (2024)
- [TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization](https://mlsanity.com/paper/2503.19901/) (2025)
- [GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling](https://mlsanity.com/paper/2501.18898/) (2025)
- [The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settings](https://mlsanity.com/paper/2308.12646/) (2023)
- [Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation](https://mlsanity.com/paper/2309.05455/) (2023)
- [Viser: Imperative, Web-based 3D Visualization in Python](https://mlsanity.com/paper/2507.22885/) (2025)
- [TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions](https://mlsanity.com/paper/2412.06334/) (2024)
- [Embody 3D: A Large-scale Multimodal Motion and Behavior Dataset](https://mlsanity.com/paper/2510.16258/) (2025)
- [PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image](https://mlsanity.com/paper/2510.11649/) (2025)
- [Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understanding](https://mlsanity.com/paper/2504.09623/) (2025)
- [ProjFlow: Projection Sampling with Flow Matching for Zero-Shot Exact Spatial Motion Control](https://mlsanity.com/paper/2602.22742/) (2026)
- [Learning to Generate Pointing Gestures in Situated Embodied Conversational Agents](https://mlsanity.com/paper/2509.12507/) (2025)
- [SceMoS: Scene-Aware 3D Human Motion Synthesis by Planning with Geometry-Grounded Tokens](https://mlsanity.com/paper/2602.20476/) (2026)
- [MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue](https://mlsanity.com/paper/2605.21796/) (2026)
- [Grounded Gesture Generation: Language, Motion, and Space](https://mlsanity.com/paper/2507.04522/) (2025)
- [SIG-Chat: Spatial Intent-Guided Conversational Gesture Generation Involving How, When and Where](https://mlsanity.com/paper/2509.23852/) (2025)

## Abstract (arXiv)

Communication in shared space interweaves verbal and non-verbal signals, and pointing gestures anchor language to the environment: "put the cup on that one" is uninterpretable without the gesture that fixes the referent. Yet no common framework exists for evaluating whether generated gestures indicate their intended referent; distributional metrics reward a gesture aimed at the wrong object as long as it looks natural. We introduce a benchmark for spatially grounded gesture generation, comprising ~2K pointing-annotated clips from naturalistic VR dialogue with ground-truth 3D referents, a task in which systems must decide when, how and where to point within conversational speech, and a protocol that separates temporal alignment, spatial grounding and perceived naturalness. We also provide a flow-matching baseline, MM-Conv-Flow. Evaluating it alongside an independent retrieval-based system and captured human motion, we find that geometric grounding can exceed that of human pointing without any gain in perceived naturalness, showing that referential gesture quality must be measured along separate dimensions.