# Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants

arXiv: https://arxiv.org/abs/2502.07956 · HTML: https://mlsanity.com/paper/2502.07956/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2025.

## Built on by (most cited first, 2 of 2)

- [DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues](https://mlsanity.com/paper/2506.22853/) (2025)
- [Evaluating the Effectiveness of Large Language Models in Solving Simple Programming Tasks: A User-Centered Study](https://mlsanity.com/paper/2507.04043/) (2025)

## Abstract (arXiv)

As Large Language Models (LLMs) are increasingly adopted in software engineering, recently in the form of conversational assistants, ensuring these technologies align with developers' needs is essential. The limitations of traditional human-centered methods for evaluating LLM-based tools at scale raise the need for automatic evaluation. In this paper, we advocate combining insights from human-computer interaction (HCI) and artificial intelligence (AI) research to enable human-centered automatic evaluation of LLM-based conversational SE assistants. We identify requirements for such evaluation and challenges down the road, working towards a framework that ensures these assistants are designed and deployed in line with user needs.