mlsanityData to 7 Oct 2026

Integrating Representational Gestures into Automatically Generated Embodied Explanations and its Effects on Understanding and Interaction Quality

arXiv 2406.12544 ↗First posted 18 Jun 2024cs.HC cs.CV cs.GRcited by 0

Not on the map. Why?

Abstract (arXiv)

In human interaction, gestures serve various functions such as marking speech rhythm, highlighting key elements, and supplementing information. These gestures are also observed in explanatory contexts. However, the impact of gestures on explanations provided by virtual agents remains underexplored. A user study was carried out to investigate how different types of gestures influence perceived interaction quality and listener understanding. This study addresses the effect of gestures in explanation by developing an embodied virtual explainer integrating both beat gestures and iconic gestures to enhance its automatically generated verbal explanations. Our model combines beat gestures generated by a learned speech-driven synthesis module with manually captured iconic gestures, supporting the agent's verbal expressions about the board game Quarto! as an explanation scenario. Findings indicate that neither the use of iconic gestures alone nor their combination with beat gestures outperforms the baseline or beat-only conditions in terms of understanding. Nonetheless, compared to prior research, the embodied agent significantly enhances understanding.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 102 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

18 of 102 references.

References · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  4. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  5. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  7. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  8. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  9. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  10. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  11. 2020Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualizationcited by 170
  12. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 128
6 more
  1. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  2. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  3. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 27
  4. 2023AQ-GT: a Temporally Aligned and Quantized GRU-Transformer for Co-Speech Gesture Synthesiscited by 8
  5. 2023Forms of Understanding for XAI-Explanationscited by 4
  6. 2023Augmented Co-Speech Gesture Generation: Including Form and Meaning Features to Guide Learning-Based Gesture Synthesiscited by 3

Built on by

Not cited yet.