# Hallmarks of Human-Machine Collaboration: A framework for assessment in the DARPA Communicating with Computers Program

arXiv: https://arxiv.org/abs/2102.04958 · HTML: https://mlsanity.com/paper/2102.04958/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 17 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 4 of 4)

- [Simulating Action Dynamics with Neural Process Networks](https://mlsanity.com/paper/1711.05313/) (2017)
- [Discourse-Aware Neural Rewards for Coherent Text Generation](https://mlsanity.com/paper/1805.03766/) (2018)
- [VoxML: A Visualization Modeling Language](https://mlsanity.com/paper/1610.01508/) (2016)
- [Composition by Conversation](https://mlsanity.com/paper/1709.02076/) (2017)

## Built on by (most cited first, 6 of 6)

- [Exploring XAI for the Arts: Explaining Latent Space in Generative Music](https://mlsanity.com/paper/2308.05496/) (2023)
- [Natural Language-conditioned Reinforcement Learning with Inside-out Task Language Development and Translation](https://mlsanity.com/paper/2302.09368/) (2023)
- [Human-Centered AI Communication in Co-Creativity: An Initial Framework and Insights](https://mlsanity.com/paper/2505.18385/) (2025)
- [Improving User Experience with FAICO: Towards a Framework for AI Communication in Human-AI Co-Creativity](https://mlsanity.com/paper/2504.02526/) (2025)
- ["How to best say it?" : Translating Directives in Machine Language into Natural Language in the Blocks World](https://mlsanity.com/paper/2107.06886/) (2021)
- [Roots and Requirements for Collaborative AIs](https://mlsanity.com/paper/2303.12040/) (2023)

## Abstract (arXiv)

There is a growing desire to create computer systems that can communicate effectively to collaborate with humans on complex, open-ended activities. Assessing these systems presents significant challenges. We describe a framework for evaluating systems engaged in open-ended complex scenarios where evaluators do not have the luxury of comparing performance to a single right answer. This framework has been used to evaluate human-machine creative collaborations across story and music generation, interactive block building, and exploration of molecular mechanisms in cancer. These activities are fundamentally different from the more constrained tasks performed by most contemporary personal assistants as they are generally open-ended, with no single correct solution, and often no obvious completion criteria. We identified the Key Properties that must be exhibited by successful systems. From there we identified "Hallmarks" of success -- capabilities and features that evaluators can observe that would be indicative of progress toward achieving a Key Property. In addition to being a framework for assessment, the Key Properties and Hallmarks are intended to serve as goals in guiding research direction.