# INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback

arXiv: https://arxiv.org/abs/2305.14282 · HTML: https://mlsanity.com/paper/2305.14282/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://mlsanity.com/paper/2004.14564/) (2020)
- [BLEU might be Guilty but References are not Innocent](https://mlsanity.com/paper/2004.06063/) (2020)
- [UniTE: Unified Translation Evaluation](https://mlsanity.com/paper/2204.13346/) (2022)
- [Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task](https://mlsanity.com/paper/2010.04297/) (2020)
- [Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis](https://mlsanity.com/paper/2210.05035/) (2022)
- [Toward Human-Like Evaluation for Natural Language Generation with Error Analysis](https://mlsanity.com/paper/2212.10179/) (2022)
- [Towards Explainable Evaluation Metrics for Natural Language Generation](https://mlsanity.com/paper/2203.11131/) (2022)

## Built on by (most cited first, 22 of 22)

- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics](https://mlsanity.com/paper/2310.19792/) (2023)
- [Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale](https://mlsanity.com/paper/2601.03444/) (2026)
- [Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation](https://mlsanity.com/paper/2401.10186/) (2024)
- [From Handcrafted Features to LLMs: A Brief Survey for Machine Translation Quality Estimation](https://mlsanity.com/paper/2403.14118/) (2024)
- [Are AI Detectors Good Enough? A Survey on Quality of Datasets With Machine-Generated Texts](https://mlsanity.com/paper/2410.14677/) (2024)
- [Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contexts](https://mlsanity.com/paper/2504.04713/) (2025)
- [AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discovery](https://mlsanity.com/paper/2504.07421/) (2025)
- [Quantitative LLM Judges](https://mlsanity.com/paper/2506.02945/) (2025)
- [Self-Taught Self-Correction for Small Language Models](https://mlsanity.com/paper/2503.08681/) (2025)
- [Digital Socrates: Evaluating LLMs through Explanation Critiques](https://mlsanity.com/paper/2311.09613/) (2023)
- [OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMs](https://mlsanity.com/paper/2503.11858/) (2025)
- [RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelines](https://mlsanity.com/paper/2505.13538/) (2025)
- [Cascaded Self-Evaluation Augmented Training for Lightweight Multimodal LLMs](https://mlsanity.com/paper/2501.05662/) (2025)
- [EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance Generation](https://mlsanity.com/paper/2508.06046/) (2025)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR](https://mlsanity.com/paper/2406.10880/) (2024)
- [Translation Canvas: An Explainable Interface to Pinpoint and Analyze Translation Systems](https://mlsanity.com/paper/2410.10861/) (2024)
- [Faster Machine Translation Ensembling with Reinforcement Learning and Competitive Correction](https://mlsanity.com/paper/2501.15219/) (2025)
- [Exploring the features used for summary evaluation by Human and GPT](https://mlsanity.com/paper/2512.19620/) (2025)
- [On the Use of LLMs for Specialised Terminology: A Good Alternative to Corpora?](https://mlsanity.com/paper/2607.24784/) (2026)

## Abstract (arXiv)

Automatically evaluating the quality of language generation is critical. Although recent learned metrics show high correlation with human judgement, these metrics can not explain their verdict or associate the scores with defects in generated text. To address this limitation, we present InstructScore, an explainable evaluation metric for text generation. By harnessing both explicit human instruction and the implicit knowledge of GPT-4, we fine-tune a text evaluation metric based on LLaMA, producing both a score for generated text and a human readable diagnostic report. We evaluate InstructScore on a variety of generation tasks, including translation, captioning, data-to-text and commonsense generation. Experiments show that our 7B model surpasses all other unsupervised metrics, including those based on 175B GPT-3 and GPT-4. Surprisingly, our InstructScore, even without direct supervision from human-rated data, achieves performance levels on par with state-of-the-art metrics like COMET22, which were fine-tuned on human ratings.