# From Metrics to Meaning: Time to Rethink Evaluation in Human-AI Collaborative Design

arXiv: https://arxiv.org/abs/2402.07911 · HTML: https://mlsanity.com/paper/2402.07911/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 44 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 12 of 12)

- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Illuminating Diverse Neural Cellular Automata for Level Generation](https://mlsanity.com/paper/2109.05489/) (2021)
- [Mapping Hearthstone Deck Spaces through MAP-Elites with Sliding Boundaries](https://mlsanity.com/paper/1904.10656/) (2019)
- [Empowering Quality Diversity in Dungeon Design with Interactive Constrained MAP-Elites](https://mlsanity.com/paper/1906.05175/) (2019)
- [Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints](https://mlsanity.com/paper/2001.05424/) (2020)
- [GUIComp: A GUI Design Assistant with Real-Time, Multi-Faceted Feedback](https://mlsanity.com/paper/2001.05684/) (2020)
- [Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniques](https://mlsanity.com/paper/2204.07641/) (2022)
- [Evaluating Mixed-Initiative Procedural Level Design Tools using a Triple-Blind Mixed-Method User Study](https://mlsanity.com/paper/2005.07478/) (2020)
- [Baba is Y'all 2.0: Design and Investigation of a Collaborative Mixed-Initiative System](https://mlsanity.com/paper/2203.02035/) (2022)
- [A Vision For Continuous Automated Game Design](https://mlsanity.com/paper/1707.09661/) (2017)
- [Learning the Designer's Preferences to Drive Evolution](https://mlsanity.com/paper/2003.03268/) (2020)
- [It's the Journey Not the Destination: Building Genetic Algorithms Practitioners Can Trust](https://mlsanity.com/paper/2010.06406/) (2020)

## Abstract (arXiv)

As AI systems increasingly shape decision making in creative design contexts, understanding how humans engage with these tools has become a critical challenge for interactive intelligent systems research. This paper contributes a challenge to rethink how to evaluate human--AI collaborative systems, advocating for a more nuanced and multidimensional approach. Findings from one of the largest field studies to date (n = 808) of a human--AI co-creative system, The Genetic Car Designer, complemented by a controlled lab study (n = 12) are presented. The system is based on an interactive evolutionary algorithm where participants were tasked with designing a simple two dimensional representation of a car. Participants were exposed to galleries of design suggestions generated by an intelligent system, MAP--Elites, and a random control. Results indicate that exposure to galleries generated by MAP--Elites significantly enhanced both cognitive and behavioural engagement, leading to higher-quality design outcomes. Crucially for the wider community, the analysis reveals that conventional evaluation methods, which often focus on solely behavioural and design quality metrics, fail to capture the full spectrum of user engagement. By considering the human--AI design process as a changing emotional, behavioural and cognitive state of the designer, we propose evaluating human--AI systems holistically and considering intelligent systems as a core part of the user experience -- not simply a back end tool.