mlsanityData to 14 Oct 2026

SPHERE: An Evaluation Card for Human-AI Systems

arXiv 2504.07971 ↗First posted 24 Mar 2025cs.HC cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

In the era of Large Language Models (LLMs), establishing effective evaluation methods and standards for diverse human-AI interaction systems is increasingly challenging. To encourage more transparent documentation and facilitate discussion on human-AI system evaluation design options, we present an evaluation card SPHERE, which encompasses five key dimensions: 1) What is being evaluated?; 2) How is the evaluation conducted?; 3) Who is participating in the evaluation?; 4) When is evaluation conducted?; 5) How is evaluation validated? We conduct a review of 39 human-AI systems using SPHERE, outlining current evaluation practices and areas for improvement. We provide three recommendations for improving the validity and rigor of evaluation practices.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 49 of its 143 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

49 of 143 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2018Datasheets for Datasetscited by 1,386
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  6. 2018Model Cards for Model Reportingcited by 1,080
  7. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  8. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 268
  9. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  10. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  11. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  12. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
37 more
  1. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
  2. 2020With Little Power Comes Great Responsibilitycited by 142
  3. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 97
  4. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 97
  5. 2022Evaluating Human-Language Model Interactioncited by 92
  6. 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
  7. 2024A Design Space for Intelligent and Interactive Writing Assistantscited by 76
  8. 2020Human Factors in Model Interpretability: Industry Practices, Challenges, and Needscited by 74
  9. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 74
  10. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  11. 2023CoEdIT: Text Editing by Task-Specific Instruction Tuningcited by 68
  12. 2023Rethinking Model Evaluation as Narrowing the Socio-Technical Gapcited by 56
  13. 2024Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approachcited by 54
  14. 2023"What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Modelscited by 48
  15. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
  16. 2023CoQuest: Exploring Research Question Co-Creation with an LLM-based Agentcited by 34
  17. 2023How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debuggingcited by 30
  18. 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 30
  19. 2023CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Toolscited by 29
  20. 2022Understanding Machine Learning Practitioners' Data Documentation Perceptions, Needs, Challenges, and Desideratacited by 28
  21. 2023ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Reviewcited by 28
  22. 2024DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialoguescited by 24
  23. 2023RELIC: Investigating Large Language Model Responses using Self-Consistencycited by 23
  24. 2023Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLPcited by 21
  25. 2024PaperWeaver: Enriching Topical Paper Alerts by Contextualizing Recommended Papers with User-collected Paperscited by 18
  26. 2024EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbotcited by 17
  27. 2023VAL: Interactive Task Learning with GPT Dialog Parsingcited by 14
  28. 2023Harnessing the Power of LLMs: Evaluating Human-AI Text Co-Creation through the Lens of News Headline Generationcited by 14
  29. 2024Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflowcited by 14
  30. 2022User-Driven Research of Medical Note Generation Softwarecited by 13
  31. 2023Low-code LLM: Graphical User Interface over Large Language Modelscited by 13
  32. 2020A Comparative Analysis of Industry Human-AI Interaction Guidelinescited by 10
  33. 2024Think Fast, Think Slow, Think Critical: Designing an Automated Propaganda Detection Toolcited by 10
  34. 2023Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMscited by 9
  35. 2024Plan-Grounded Large Language Models for Dual Goal Conversational Settingscited by 8
  36. 2024Proofread: Fixes All Errors with One Tapcited by 4
  37. 2024Generating Signed Language Instructions in Large-Scale Dialogue Systemscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 29
  2. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  3. 2025Embracing Contradiction: Theoretical Inconsistency Will Not Impede the Road of Building Responsible AI Systemscited by 8
  4. 2026PrefIx: Understand and Adapt to User Preference in Human-Agent Interactioncited by 7
  5. 2026Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluationcited by 5
  6. 2025EvalCards: A Framework for Standardized Evaluation Reportingcited by 4
  7. 2025Social Welfare Function Leaderboard: When LLM Agents Allocate Social Welfarecited by 2
  8. 2026Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluationcited by 0
  9. 2026Breakdowns in Conversational AI: Interactional Failures in Emotionally and Ethically Sensitive Contextscited by 0