mlsanityData to 14 Oct 2026

The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation

arXiv 2610.03436 ↗First posted 2 Oct 2026cs.GR cs.CV cs.MMcited by 0

Not on the map. Why?

Abstract (arXiv)

Speech-driven 3D facial animation can reproduce recognizable mouth poses. However, it can simplify the motion between them, and that motion carries coarticulation, the way the sounds around each sound shape its articulation. We introduce a geometric measure of this trajectory shaping: lip-path length compared with the shortest route through the vowel, consonant and vowel positions of a speech segment. In contrast to the endpoint chord, this consonant-aware route accounts for obligatory transit and avoids degeneracy, while preserving invariance to uniform motion gain. The measure needs only a forced alignment, so it applies where no ground truth exists. We demonstrate it on four state-of-the-art methods, one per architectural family, real-time and offline. All four trace flatter lip trajectories than captured speech. Against frame-rate-matched ground truth, DiffPoseTalk, ARTalk and FaceFormer show clear deficits, equivalent on this measure to removing 15-60% of real speech's fast articulatory component. CodeTalker is marginal on the primary measure and clear on a companion measure. A pre-registered study with 97 viewers and 3,523 judgments underpins the measured direction: controlled damping of real motion lowers the score and is penalized, whereas exaggeration shows no detected penalty over the tested range. Viewers also prefer real speech in 73.4% of sentence comparisons and, in the aggregate, on single words. Together, the measure, its calibration and the study identify a perceptually relevant loss of trajectory shaping and a concrete target for improving synthesized articulation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 42 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

14 of 42 references.

References · 14

  1. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  2. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 259
  3. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  4. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
  5. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 142
  6. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 78
  7. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 74
  8. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 68
  9. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 30
  10. 2025ARTalk: Speech-Driven 3D Head Animation via Autoregressive Modelcited by 29
  11. 2024ScanTalk: 3D Talking Heads from Unregistered Scanscited by 24
  12. 2025Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metricscited by 10
2 more
  1. 2024Beyond Fixed Topologies: Unregistered Training and Comprehensive Evaluation Metrics for 3D Talking Headscited by 5
  2. 2023Perception of Human Motion with Different Geometric Modelscited by 1

Built on by

Not cited yet.