# Do Motion Tokenizers for Co-Speech Gesture Generation Encode Gesture Semantics?

arXiv: https://arxiv.org/abs/2610.03765 · HTML: https://mlsanity.com/paper/2610.03765/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling](https://mlsanity.com/paper/2401.00374/) (2023)
- [Analyzing Input and Output Representations for Speech-Driven Gesture Generation](https://mlsanity.com/paper/1903.03369/) (2019)
- [Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis](https://mlsanity.com/paper/2405.09814/) (2024)
- [Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis](https://mlsanity.com/paper/2412.06786/) (2024)
- [Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importance](https://mlsanity.com/paper/2010.00995/) (2020)
- [Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation](https://mlsanity.com/paper/2409.10535/) (2024)
- [GestureCoach: Rehearsing for Engaging Talks with LLM-Driven Gesture Recommendations](https://mlsanity.com/paper/2504.10706/) (2025)
- [Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures](https://mlsanity.com/paper/2605.30608/) (2026)

## Abstract (arXiv)

Discrete motion tokenizers encode motion as atomic units and are widely used for co-speech gesture generation. It remains unclear which motion properties, especially those relevant to gesture semantics, are recoverable from these codebooks. We probe a reconstruction-trained codebook using 19 co-speech gesture descriptors spanning from raw motion to abstract communicative function. Results show that geometry and handedness are readily decodable from token embeddings, while motion category is only weakly decoded despite showing systematic differences in discrete code usage. This gap between reconstruction quality and descriptor decodability suggests that reconstruction objectives alone do not guarantee that gesture semantics are captured, and that evaluating codebooks on such properties can guide the design of more semantic motion tokenizers.