mlsanityData to 14 Oct 2026

Towards Better User Studies in Computer Graphics and Vision

arXiv 2206.11461 ↗First posted 23 Jun 2022cs.GR cs.CV cs.HCcited by 11

Not on the map. Why?

Abstract (arXiv)

Online crowdsourcing platforms have made it increasingly easy to perform evaluations of algorithm outputs with survey questions like "which image is better, A or B?", leading to their proliferation in vision and graphics research papers. Results of these studies are often used as quantitative evidence in support of a paper's contributions. On the one hand we argue that, when conducted hastily as an afterthought, such studies lead to an increase of uninformative, and, potentially, misleading conclusions. On the other hand, in these same communities, user research is underutilized in driving project direction and forecasting user needs and reception. We call for increased attention to both the design and reporting of user studies in computer vision and graphics papers towards (1) improved replicability and (2) improved project direction. Together with this call, we offer an overview of methodologies from user experience research (UXR), human-computer interaction (HCI), and applied perception to increase exposure to the available methodologies and best practices. We discuss foundational user research methods (e.g., needfinding) that are presently underutilized in computer vision and graphics research, but can provide valuable project direction. We provide further pointers to the literature for readers interested in exploring other UXR methodologies. Finally, we describe broader open issues and recommendations for the research community.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 11 of its 116 references that are in the map. First posted on arXiv 23 Jun 2022.

Builds on

11 of 116 references.

References · 11

  1. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
  2. 2015Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networkscited by 214
  3. 2021The Benchmark Lotterycited by 136
  4. 2018A Closed-form Solution to Photorealistic Image Stylizationcited by 133
  5. 2018Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Modelscited by 106
  6. 2019HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Modelscited by 59
  7. 2016Crowdsourcing in Computer Visioncited by 31
  8. 2020Predicting Visual Importance Across Graphic Design Typescited by 30
  9. 2018Terminologies for Reproducible Researchcited by 11
  10. 2021Countering Racial Bias in Computer Graphics Researchcited by 8
  11. 2020Code Replicability in Computer Graphicscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 4; 2025: 3.

11 papers.

  1. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  2. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 101
  3. 2023CityDreamer: Compositional Generative Model of Unbounded 3D Citiescited by 76
  4. 2023SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collectionscited by 72
  5. 2024Generative Gaussian Splatting for Unbounded 3D City Generationcited by 44
  6. 2023How Will It Drape Like? Capturing Fabric Mechanics from Depth Imagescited by 6
  7. 2024Evaluation in Neural Style Transfer: A Reviewcited by 4
  8. 2025Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmarkcited by 1
  9. 2023Composite Diffusion | whole >= Sigma partscited by 0
  10. 2025WebXAII: an open-source web framework to study human-XAI interactioncited by 0
  11. 2025AnchorHOI: Zero-shot Generation of 4D Human-Object Interaction via Anchor-based Prior Distillationcited by 0