# The Impact of Expertise in the Loop for Exploring Machine Rationality

arXiv: https://arxiv.org/abs/2302.05665 · HTML: https://mlsanity.com/paper/2302.05665/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 75 references that are in the map.
- First posted on arXiv 11 Feb 2023.

## Builds on (most cited first, 15 of 15)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Expanding Explainability: Towards Social Transparency in AI systems](https://mlsanity.com/paper/2101.04719/) (2021)
- [Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement](https://mlsanity.com/paper/2105.08195/) (2021)
- [Constrained Bayesian Optimization with Noisy Experiments](https://mlsanity.com/paper/1706.07094/) (2017)
- [Preferential Bayesian Optimization](https://mlsanity.com/paper/1704.03651/) (2017)
- [The reparameterization trick for acquisition functions](https://mlsanity.com/paper/1712.00424/) (2017)
- [Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes](https://mlsanity.com/paper/2203.11382/) (2022)
- [Sequential Gallery for Interactive Visual Design Optimization](https://mlsanity.com/paper/2005.04107/) (2020)
- [Projective Preferential Bayesian Optimization](https://mlsanity.com/paper/2002.03113/) (2020)
- [Preferential Batch Bayesian Optimization](https://mlsanity.com/paper/2003.11435/) (2020)
- [Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniques](https://mlsanity.com/paper/2204.07641/) (2022)
- [Interactive Text Ranking with Bayesian Optimisation: A Case Study on Community QA and Summarisation](https://mlsanity.com/paper/1911.10183/) (2019)
- [The Human in the Infinite Loop: A Case Study on Revealing and Explaining Human-AI Interaction Loop Failures](https://mlsanity.com/paper/2207.12761/) (2022)
- [Trust in Prediction Models: a Mixed-Methods Pilot Study on the Impact of Domain Expertise](https://mlsanity.com/paper/2109.08183/) (2021)
- [Choice functions based multi-objective Bayesian optimisation](https://mlsanity.com/paper/2110.08217/) (2021)

## Built on by (most cited first, 1 of 1)

- [Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization](https://mlsanity.com/paper/2610.14757/) (2026)

## Abstract (arXiv)

Human-in-the-loop optimization utilizes human expertise to guide machine optimizers iteratively and search for an optimal solution in a solution space. While prior empirical studies mainly investigated novices, we analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We conducted a study (N=60) in text, photo, and 3D mesh optimization contexts. We found that novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization iteration with more explicit preference while keeping satisfaction low. In contrast, novices were more easily satisfied and terminated faster. Therefore, we identified that experts seek more diverse outcomes while the machine reaches optimal results, and the observed behavior can be used as a performance indicator for human-in-the-loop system designers to improve underlying models. We inform future research to be cautious about the impact of user expertise when designing human-in-the-loop systems.