# Towards Better User Studies in Computer Graphics and Vision

arXiv: https://arxiv.org/abs/2206.11461 · HTML: https://mlsanity.com/paper/2206.11461/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 116 references that are in the map.
- First posted on arXiv 23 Jun 2022.

## Builds on (most cited first, 11 of 11)

- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks](https://mlsanity.com/paper/1506.05751/) (2015)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [A Closed-form Solution to Photorealistic Image Stylization](https://mlsanity.com/paper/1802.06474/) (2018)
- [Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models](https://mlsanity.com/paper/1811.12463/) (2018)
- [HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models](https://mlsanity.com/paper/1904.01121/) (2019)
- [Crowdsourcing in Computer Vision](https://mlsanity.com/paper/1611.02145/) (2016)
- [Predicting Visual Importance Across Graphic Design Types](https://mlsanity.com/paper/2008.02912/) (2020)
- [Terminologies for Reproducible Research](https://mlsanity.com/paper/1802.03311/) (2018)
- [Countering Racial Bias in Computer Graphics Research](https://mlsanity.com/paper/2103.15163/) (2021)
- [Code Replicability in Computer Graphics](https://mlsanity.com/paper/2005.00554/) (2020)

## Built on by (most cited first, 11 of 11)

- [RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion](https://mlsanity.com/paper/2404.07199/) (2024)
- [3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion](https://mlsanity.com/paper/2409.12957/) (2024)
- [CityDreamer: Compositional Generative Model of Unbounded 3D Cities](https://mlsanity.com/paper/2309.00610/) (2023)
- [SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections](https://mlsanity.com/paper/2302.01330/) (2023)
- [Generative Gaussian Splatting for Unbounded 3D City Generation](https://mlsanity.com/paper/2406.06526/) (2024)
- [How Will It Drape Like? Capturing Fabric Mechanics from Depth Images](https://mlsanity.com/paper/2304.06704/) (2023)
- [Evaluation in Neural Style Transfer: A Review](https://mlsanity.com/paper/2401.17109/) (2024)
- [Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark](https://mlsanity.com/paper/2511.01233/) (2025)
- [Composite Diffusion | whole >= Sigma parts](https://mlsanity.com/paper/2307.13720/) (2023)
- [WebXAII: an open-source web framework to study human-XAI interaction](https://mlsanity.com/paper/2506.14777/) (2025)
- [AnchorHOI: Zero-shot Generation of 4D Human-Object Interaction via Anchor-based Prior Distillation](https://mlsanity.com/paper/2512.14095/) (2025)

## Abstract (arXiv)

Online crowdsourcing platforms have made it increasingly easy to perform evaluations of algorithm outputs with survey questions like "which image is better, A or B?", leading to their proliferation in vision and graphics research papers. Results of these studies are often used as quantitative evidence in support of a paper's contributions. On the one hand we argue that, when conducted hastily as an afterthought, such studies lead to an increase of uninformative, and, potentially, misleading conclusions. On the other hand, in these same communities, user research is underutilized in driving project direction and forecasting user needs and reception. We call for increased attention to both the design and reporting of user studies in computer vision and graphics papers towards (1) improved replicability and (2) improved project direction. Together with this call, we offer an overview of methodologies from user experience research (UXR), human-computer interaction (HCI), and applied perception to increase exposure to the available methodologies and best practices. We discuss foundational user research methods (e.g., needfinding) that are presently underutilized in computer vision and graphics research, but can provide valuable project direction. We provide further pointers to the literature for readers interested in exploring other UXR methodologies. Finally, we describe broader open issues and recommendations for the research community.