mlsanityData to 14 Oct 2026

INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback

arXiv 2305.14282 ↗First posted 23 May 2023cs.CL cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Automatically evaluating the quality of language generation is critical. Although recent learned metrics show high correlation with human judgement, these metrics can not explain their verdict or associate the scores with defects in generated text. To address this limitation, we present InstructScore, an explainable evaluation metric for text generation. By harnessing both explicit human instruction and the implicit knowledge of GPT-4, we fine-tune a text evaluation metric based on LLaMA, producing both a score for generated text and a human readable diagnostic report. We evaluate InstructScore on a variety of generation tasks, including translation, captioning, data-to-text and commonsense generation. Experiments show that our 7B model surpasses all other unsupervised metrics, including those based on 175B GPT-3 and GPT-4. Surprisingly, our InstructScore, even without direct supervision from human-rated data, achieves performance levels on par with state-of-the-art metrics like COMET22, which were fine-tuned on human ratings.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 23 of its 53 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

23 of 53 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  10. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  12. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
11 more
  1. 2023GPTScore: Evaluate as You Desirecited by 361
  2. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  3. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  4. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 214
  5. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
  6. 2020BLEU might be Guilty but References are not Innocentcited by 110
  7. 2022UniTE: Unified Translation Evaluationcited by 28
  8. 2020Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Taskcited by 25
  9. 2022Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesiscited by 18
  10. 2022Toward Human-Like Evaluation for Natural Language Generation with Error Analysiscited by 18
  11. 2022Towards Explainable Evaluation Metrics for Natural Language Generationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 10; 2026: 3.

22 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  3. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  4. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scalecited by 20
  5. 2024Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generationcited by 16
  6. 2024From Handcrafted Features to LLMs: A Brief Survey for Machine Translation Quality Estimationcited by 10
  7. 2024Are AI Detectors Good Enough? A Survey on Quality of Datasets With Machine-Generated Textscited by 10
  8. 2025Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contextscited by 9
  9. 2025AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discoverycited by 6
  10. 2025Quantitative LLM Judgescited by 6
  11. 2025Self-Taught Self-Correction for Small Language Modelscited by 5
  12. 2023Digital Socrates: Evaluating LLMs through Explanation Critiquescited by 4
10 more
  1. 2025OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMscited by 4
  2. 2025RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelinescited by 4
  3. 2025Cascaded Self-Evaluation Augmented Training for Lightweight Multimodal LLMscited by 2
  4. 2025EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance Generationcited by 2
  5. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  6. 2024Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASRcited by 1
  7. 2024Translation Canvas: An Explainable Interface to Pinpoint and Analyze Translation Systemscited by 1
  8. 2025Faster Machine Translation Ensembling with Reinforcement Learning and Competitive Correctioncited by 1
  9. 2025Exploring the features used for summary evaluation by Human and GPTcited by 0
  10. 2026On the Use of LLMs for Specialised Terminology: A Good Alternative to Corpora?cited by 0