mlsanityData to 14 Oct 2026

ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?

arXiv 2403.17368 ↗First posted 26 Mar 2024cs.CL cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

As AI becomes more integral in our lives, the need for transparency and responsibility grows. While natural language explanations (NLEs) are vital for clarifying the reasoning behind AI decisions, evaluating them through human judgments is complex and resource-intensive due to subjectivity and the need for fine-grained ratings. This study explores the alignment between ChatGPT and human assessments across multiple scales (i.e., binary, ternary, and 7-Likert scale). We sample 300 data instances from three NLE datasets and collect 900 human annotations for both informativeness and clarity scores as the text quality measurement. We further conduct paired comparison experiments under different ranges of subjectivity scores, where the baseline comes from 8,346 human annotations. Our results show that ChatGPT aligns better with humans in more coarse-grained scales. Also, paired comparisons and dynamic prompting (i.e., providing semantically similar examples in the prompt) improve the alignment. This research advances our understanding of large language models' capabilities to assess the text explanation quality in different configurations for responsible AI development.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 53 of its 87 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

53 of 87 references.

References · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  10. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  11. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
38 more
  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  2. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  3. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 481
  4. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  5. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  6. 2023Can AI-Generated Text be Reliably Detected?cited by 386
  7. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  8. 2016Solving General Arithmetic Word Problemscited by 360
  9. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
  10. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  11. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 217
  12. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  13. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 166
  14. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
  15. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  16. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 103
  17. 2020NILE : Natural Language Inference with Faithful Natural Language Explanationscited by 96
  18. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  19. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 89
  20. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  21. 2019Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanationscited by 71
  22. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 69
  23. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  24. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  25. 2023Evaluating GPT-3 Generated Explanations for Hateful Content Moderationcited by 46
  26. 2021A Study of Automatic Metrics for the Evaluation of Natural Language Explanationscited by 34
  27. 2023Superhuman Artificial Intelligence Can Improve Human Decision Making by Increasing Noveltycited by 27
  28. 2022Chain of Explanation: New Prompting Method to Generate Higher Quality Natural Language Explanation for Implicit Hate Speechcited by 25
  29. 2022Few-shot Reranking for Multi-hop QA via Language Model Promptingcited by 19
  30. 2021Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoningcited by 18
  31. 2022ALERT: Adapting Language Models to Reasoning Taskscited by 17
  32. 2021Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setupcited by 16
  33. 2022Towards Responsible Natural Language Annotation for the Varieties of Arabiccited by 14
  34. 2023YouNICon: YouTube's CommuNIty of Conspiracy Videoscited by 13
  35. 2023Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanationscited by 13
  36. 2023Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote methodcited by 10
  37. 2022Transparency Helps Reveal When Language Models Learn Meaningcited by 8
  38. 2024ToBlend: Token-Level Blending With an Ensemble of LLMs to Attack AI-Generated Text Detectioncited by 6

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  2. 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 57
  3. 2024Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandemcited by 9
  4. 2024Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaborationcited by 8
  5. 2025PropXplain: Can LLMs Enable Explainable Propaganda Detection?cited by 8
  6. 2026Can Large Language Models Still Explain Themselves? Investigating the Impact of Quantization on Self-Explanationscited by 5
  7. 2025SEFL: A Framework for Generating Synthetic Educational Assignment Feedback with LLM Agentscited by 3
  8. 2024What Would You Ask When You First Saw a2+b2=c2? Evaluating LLM on Curiosity-Driven Questioningcited by 1
  9. 2025Aligning Black-box Language Models with Human Judgmentscited by 1
  10. 2025Corrections Meet Explanations: A Unified Framework for Explainable Grammatical Error Correctioncited by 1
  11. 2025Explaining Sources of Uncertainty in Automated Fact-Checkingcited by 1
  12. 2025LaajMeter: A Framework for LaaJ Evaluationcited by 1
5 more
  1. 2026iFlip: Iterative Feedback-driven Counterfactual Example Refinementcited by 1
  2. 2026Vulnerability of LLMs' Stated Beliefs? LLMs Belief Resistance Check Through Strategic Persuasive Conversation Interventionscited by 1
  3. 2026Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainabilitycited by 1
  4. 2025How Do Hackathons Foster Creativity? Towards AI Collaborative Evaluation of Creativity at Scalecited by 0
  5. 2025NativQA Framework: Enabling LLMs and VLMs with Native, Local, and Everyday Knowledgecited by 0