Reproducible Subjective Evaluation
Not on the map. Why?
Abstract (arXiv)
Human perceptual studies are the gold standard for the evaluation of many research tasks in machine learning, linguistics, and psychology. However, these studies require significant time and cost to perform. As a result, many researchers use objective measures that can correlate poorly with human evaluation. When subjective evaluations are performed, they are often not reported with sufficient detail to ensure reproducibility. We propose Reproducible Subjective Evaluation (ReSEval), an open-source framework for quickly deploying crowdsourced subjective evaluations directly from Python. ReSEval lets researchers launch A/B, ABX, Mean Opinion Score (MOS) and MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA) tests on audio, image, text, or video data from a command-line interface or using one line of Python, making it as easy to run as objective evaluation. With ReSEval, researchers can reproduce each other's subjective evaluations by sharing a configuration file and the audio, image, text, or video files.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 9 of its 25 references that are in the map. First posted on arXiv 8 Mar 2022.
Builds on
9 of 25 references.
References · 9
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 2021Zero-Shot Text-to-Image Generationcited by 4,288
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
- 2017Why We Need New Evaluation Metrics for NLGcited by 233
- 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
- 2019High Fidelity Speech Synthesis with Adversarial Networkscited by 92
- 2021GAN-Control: Explicitly Controllable GANscited by 91
- 2021CDPAM: Contrastive learning for perceptual audio similaritycited by 42
- 2021Chunked Autoregressive GAN for Conditional Waveform Synthesiscited by 38
Built on by
Citing papers per year: 2022: 1; 2023: 1.
2 papers.