mlsanityData to 7 Oct 2026

Evaluating Human-Language Model Interaction

arXiv 2212.09746 ↗First posted 19 Dec 2022cs.CLcited by 90

Not on the map. Why?

Abstract (arXiv)

Many real-world applications of language models (LMs), such as writing assistance and code autocomplete, involve human-LM interaction. However, most benchmarks are non-interactive in that a model produces output without human involvement. To evaluate human-LM interaction, we develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and dimensions to consider when designing evaluation metrics. Compared to standard, non-interactive evaluation, HALIE captures (i) the interactive process, not only the final output; (ii) the first-person subjective experience, not just a third-party assessment; and (iii) notions of preference beyond quality (e.g., enjoyment and ownership). We then design five tasks to cover different forms of interaction: social dialogue, question answering, crossword puzzles, summarization, and metaphor generation. With four state-of-the-art LMs (three variants of OpenAI's GPT-3 and AI21 Labs' Jurassic-1), we find that better non-interactive performance does not always translate to better human-LM interaction. In particular, we highlight three cases where the results from non-interactive and interactive metrics diverge and underscore the importance of human-LM interaction for LM evaluation.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Builds on 90 of its 207 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

90 of 207 references.

References · 90

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2020Learning to summarize from human feedbackcited by 2,562
38 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  13. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  14. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  15. 2020Recipes for building an open-domain chatbotcited by 593
  16. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  17. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  18. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  19. 2016Learning to reinforcement learncited by 369
  20. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  21. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  22. 2016Cooperative Inverse Reinforcement Learningcited by 298
  23. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  24. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  25. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  26. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  27. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  28. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  29. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  30. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  31. 2019Challenges in Building Intelligent Open-domain Dialog Systemscited by 147
  32. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  33. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  34. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  35. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  36. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  37. 2019Learning to Speak and Act in a Fantasy Text Adventure Gamecited by 122
  38. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 45; 2024: 19; 2025: 17; 2026: 9.

90 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  5. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  6. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  7. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  8. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 195
  9. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  10. 2023GPT detectors are biased against non-native English writerscited by 175
  11. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  12. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
38 more
  1. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  2. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  3. 2024Building Machines that Learn and Think with Peoplecited by 73
  4. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  5. 2023Prompted LLMs as Chatbot Modules for Long Open-domain Conversationcited by 61
  6. 2023Facilitating Self-Guided Mental Health Interventions Through Human-Language Model Interaction: A Case Study of Cognitive Restructuringcited by 53
  7. 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
  8. 2023Opportunities and Risks of LLMs for Scalable Deliberation with Poliscited by 41
  9. 2023LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzlescited by 41
  10. 2023Beyond the Chat: Executable and Verifiable Text-Editing with LLMscited by 35
  11. 2023Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writerscited by 32
  12. 2023Ecosystem Graphs: The Social Footprint of Foundation Modelscited by 31
  13. 2023Interactive Natural Language Processingcited by 31
  14. 2023Benchmarking Large Language Model Capabilities for Conditional Generationcited by 29
  15. 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 28
  16. 2024Towards interactive evaluations for interaction harms in human-AI systemscited by 27
  17. 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 27
  18. 2025Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Modelscited by 25
  19. 2024Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Functioncited by 24
  20. 2025On Benchmarking Human-Like Intelligence in Machinescited by 24
  21. 2023Fluid Transformers and Creative Analogies: Exploring Large Language Models' Capacity for Augmenting Cross-Domain Analogical Creativitycited by 23
  22. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  23. 2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modellingcited by 21
  24. 2023Impact of Guidance and Interaction Strategies for LLM Use on Learner Performance and Perceptioncited by 19
  25. 2024IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answeringcited by 18
  26. 2023ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Modelscited by 17
  27. 2023User Experience Design Professionals' Perceptions of Generative Artificial Intelligencecited by 16
  28. 2025Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Reviewcited by 16
  29. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  30. 2023Learning Personalized Decision Support Policiescited by 13
  31. 2023Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanationscited by 13
  32. 2025Generative Value Conflicts Reveal LLM Prioritiescited by 13
  33. 2024Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Plannercited by 12
  34. 2023Approximating Online Human Evaluation of Social Chatbots with Promptingcited by 11
  35. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  36. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Codingcited by 11
  37. 2023RoleCraft-GLM: Advancing Personalized Role-Playing in Large Language Modelscited by 9
  38. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9

50 most cited of 90 All 90 (JSON)