# Expressive Communication: A Common Framework for Evaluating Developments in Generative Models and Steering Interfaces

arXiv: https://arxiv.org/abs/2111.14951 · HTML: https://mlsanity.com/paper/2111.14951/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 13 of 13)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [This Time with Feeling: Learning Expressive Musical Performance](https://mlsanity.com/paper/1808.03715/) (2018)
- [AI Song Contest: Human-AI Co-Creation in Songwriting](https://mlsanity.com/paper/2010.05388/) (2020)
- [Evaluating the Interpretability of Generative Models by Interactive Reconstruction](https://mlsanity.com/paper/2102.01264/) (2021)
- [Fixing a Broken ELBO](https://mlsanity.com/paper/1711.00464/) (2017)
- [Pragmatic Image Compression for Human-in-the-Loop Decision-Making](https://mlsanity.com/paper/2108.04219/) (2021)
- [Learning via social awareness: Improving a deep generative sketching model with facial feedback](https://mlsanity.com/paper/1802.04877/) (2018)

## Built on by (most cited first, 3 of 3)

- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Audio Prompt Adapter: Unleashing Music Editing Abilities for Text-to-Music with Lightweight Finetuning](https://mlsanity.com/paper/2407.16564/) (2024)
- [Designing Human-AI Collaboration to Support Learning in Counterspeech Writing](https://mlsanity.com/paper/2410.03032/) (2024)

## Abstract (arXiv)

There is an increasing interest from ML and HCI communities in empowering creators with better generative models and more intuitive interfaces with which to control them. In music, ML researchers have focused on training models capable of generating pieces with increasing long-range structure and musical coherence, while HCI researchers have separately focused on designing steering interfaces that support user control and ownership. In this study, we investigate through a common framework how developments in both models and user interfaces are important for empowering co-creation where the goal is to create music that communicates particular imagery or ideas (e.g., as is common for other purposeful tasks in music creation like establishing mood or creating accompanying music for another media). Our study is distinguished in that it measures communication through both composer's self-reported experiences, and how listeners evaluate this communication through the music. In an evaluation study with 26 composers creating 100+ pieces of music and listeners providing 1000+ head-to-head comparisons, we find that more expressive models and more steerable interfaces are important and complementary ways to make a difference in composers communicating through music and supporting their creative empowerment.