# HEMVIP: Human Evaluation of Multiple Videos in Parallel

arXiv: https://arxiv.org/abs/2101.11898 · HTML: https://mlsanity.com/paper/2101.11898/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2021.

## Built on by (most cited first, 13 of 13)

- [ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech](https://mlsanity.com/paper/2209.07556/) (2022)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation](https://mlsanity.com/paper/2208.10441/) (2022)
- [Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022](https://mlsanity.com/paper/2303.08737/) (2023)
- [The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settings](https://mlsanity.com/paper/2308.12646/) (2023)
- [Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation](https://mlsanity.com/paper/2309.05455/) (2023)
- [Integrated Speech and Gesture Synthesis](https://mlsanity.com/paper/2108.11436/) (2021)
- [The ReprGesture entry to the GENEA Challenge 2022](https://mlsanity.com/paper/2208.12133/) (2022)
- [Co-Speech Gesture Synthesis using Discrete Gesture Token Learning](https://mlsanity.com/paper/2303.12822/) (2023)
- [Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agent](https://mlsanity.com/paper/2311.12804/) (2023)
- [Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis](https://mlsanity.com/paper/2410.06327/) (2024)
- [Gesture Evaluation in Virtual Reality](https://mlsanity.com/paper/2509.12816/) (2025)
- [Conveying Meaning through Gestures: An Investigation into Semantic Co-Speech Gesture Generation](https://mlsanity.com/paper/2510.17599/) (2025)

## Abstract (arXiv)

In many research areas, for example motion and gesture generation, objective measures alone do not provide an accurate impression of key stimulus traits such as perceived quality or appropriateness. The gold standard is instead to evaluate these aspects through user studies, especially subjective evaluations of video stimuli. Common evaluation paradigms either present individual stimuli to be scored on Likert-type scales, or ask users to compare and rate videos in a pairwise fashion. However, the time and resources required for such evaluations scale poorly as the number of conditions to be compared increases. Building on standards used for evaluating the quality of multimedia codecs, this paper instead introduces a framework for granular rating of multiple comparable videos in parallel. This methodology essentially analyses all condition pairs at once. Our contributions are 1) a proposed framework, called HEMVIP, for parallel and granular evaluation of multiple video stimuli and 2) a validation study confirming that results obtained using the tool are in close agreement with results of prior studies using conventional multiple pairwise comparisons.