mlsanityData to 14 Oct 2026

Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization

arXiv 2610.14757 ↗First posted 10 Oct 2026cs.HC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Human-in-the-loop optimization refers to systems that present choices to a human decision-maker and query their judgment to optimize system parameters iteratively. The user interface typically provides either an absolute rating scale or asks for a preferred option. However, these approaches can potentially suffer from calibration problems, inconsistencies, and completeness violations of user preferences in such a sequential evaluation. This paper revisits the design space of opinion measurement interfaces for human-in-the-loop optimization. We conducted a study (N=360) using text summarization and image color enhancement tasks to evaluate human sequential feedback in six representative opinion measurement interfaces. Based on our analysis, we recommend using a listwise approach to support sequential opinion measurement, which counterweighs individual rating and preferential choice limitations and conveys more information for subsequent use. We further discuss the trade-offs in different user interface designs and provide guidelines to inform future research.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 75 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

22 of 75 references.

References · 22

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  5. 2020Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimizationcited by 179
  6. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  7. 2017Constrained Bayesian Optimization with Noisy Experimentscited by 104
  8. 2017Preferential Bayesian Optimizationcited by 74
  9. 2017The reparameterization trick for acquisition functionscited by 63
  10. 2022How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Modelscited by 44
  11. 2022Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomescited by 42
  12. 2020Sequential Gallery for Interactive Visual Design Optimizationcited by 25
10 more
  1. 2020Projective Preferential Bayesian Optimizationcited by 22
  2. 2020Top-k Ranking Bayesian Optimizationcited by 22
  3. 2020Preferential Batch Bayesian Optimizationcited by 19
  4. 2014Cost-Effective HITs for Relative Similarity Comparisonscited by 14
  5. 2019Learning Context-Dependent Choice Functionscited by 7
  6. 2022Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniquescited by 6
  7. 2019Interactive Text Ranking with Bayesian Optimisation: A Case Study on Community QA and Summarisationcited by 5
  8. 2022The Human in the Infinite Loop: A Case Study on Revealing and Explaining Human-AI Interaction Loop Failurescited by 4
  9. 2021Choice functions based multi-objective Bayesian optimisationcited by 2
  10. 2023The Impact of Expertise in the Loop for Exploring Machine Rationalitycited by 1

Built on by

Not cited yet.