mlsanityData to 7 Oct 2026

Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation

arXiv 2409.07355 ↗First posted 11 Sep 2024cs.CLcited by 12

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

This study introduces InteractEval, a framework that integrates human expertise and Large Language Models (LLMs) using the Think-Aloud (TA) method to generate attributes for checklist-based text evaluation. By combining human flexibility and reasoning with LLM consistency, InteractEval outperforms traditional non-LLM-based and LLM-based baselines across four distinct dimensions, consisting of Coherence, Fluency, Consistency, and Relevance. The experiment also investigates the effectiveness of the TA method, showing that it promotes divergent thinking in both humans and LLMs, leading to the generation of a wider range of relevant attributes and enhance text evaluation performance. Comparative analysis reveals that humans excel at identifying attributes related to internal quality (Coherence and Fluency), but LLMs perform better at those attributes related to external alignment (Consistency and Relevance). Consequently, leveraging both humans and LLMs together produces the best evaluation outcomes. In other words, this study emphasizes the necessity of effectively combining humans and LLMs in an automated checklist-based text evaluation framework. The code is available at textbfurlhttps://github.com/BBeeChu/InteractEval.git.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 34 of its 92 references that are in the map. First posted on arXiv 11 Sep 2024.

Builds on

34 of 92 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  10. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  12. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
21 more
  1. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  2. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  3. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  4. 2023GPTScore: Evaluate as You Desirecited by 357
  5. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  6. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  7. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  8. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  9. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 91
  10. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  11. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
  12. 2022PEER: A Collaborative Language Modelcited by 86
  13. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  14. 2024The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Modelscited by 62
  15. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  16. 2020GRUEN for Evaluating Linguistic Quality of Generated Textcited by 37
  17. 2024Towards Optimizing the Costs of LLM Usagecited by 33
  18. 2024DEBATE: Devil's Advocate-Based Assessment and Text Evaluationcited by 16
  19. 2024Navigating the Path of Writing: Outline-guided Text Generation with Large Language Modelscited by 15
  20. 2019Why Didn't You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Modelscited by 8
  21. 2023Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT for Mining Insights at Scalecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2026SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarioscited by 8
  2. 2025GALA: Can Graph-Augmented Large Language Model Agentic Workflows Elevate Root Cause Analysis?cited by 5
  3. 2026AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judgecited by 5
  4. 2025From Feedback to Checklists: Grounded Evaluation of AI-Generated Clinical Notescited by 4
  5. 2026FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedbackcited by 4
  6. 2025Integrating Expert Knowledge into Logical Programs via LLMscited by 3
  7. 2026From Answer Givers to Design Mentors: Guiding LLMs with the Cognitive Apprenticeship Modelcited by 1
  8. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 1
  9. 2026An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systemscited by 1
  10. 2026GALA: Graph-Augmented LLM Agents for Root Cause Analysis and Incident Response in Microservicescited by 1
  11. 2026Grounded Checklist Partial Credit for Agent Skill Trajectoriescited by 1
  12. 2025Towards Personalized Explanations for Health Simulations: A Mixed-Methods Framework for Stakeholder-Centric Summarizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Closer Look into Automatic Evaluation Using Large Language Models
  2. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  3. 2023TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs
  4. 2024ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models
  5. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  6. 2024Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences
  7. 2025Analysis of instruction-based LLMs' capabilities to score and judge text-input problems in an academic setting
  8. 2024Enhancing LLM Evaluations: The Garbling Trick