# Identifying Reliable Evaluation Metrics for Scientific Text Revision

arXiv: https://arxiv.org/abs/2506.04772 · HTML: https://mlsanity.com/paper/2506.04772/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 22 of 22)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Fill in the BLANC: Human-free quality estimation of document summaries](https://mlsanity.com/paper/2002.09836/) (2020)
- [Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration](https://mlsanity.com/paper/2305.14324/) (2023)
- [CoEdIT: Text Editing by Task-Specific Instruction Tuning](https://mlsanity.com/paper/2305.09857/) (2023)
- [Read, Revise, Repeat: A System Demonstration for Human-in-the-loop Iterative Text Revision](https://mlsanity.com/paper/2204.03685/) (2022)
- [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021)
- [EditEval: An Instruction-Based Benchmark for Text Improvements](https://mlsanity.com/paper/2209.13331/) (2022)
- [Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond](https://mlsanity.com/paper/2205.11484/) (2022)
- [Improving Iterative Text Revision by Learning Where to Edit from Other Revision Tasks](https://mlsanity.com/paper/2212.01350/) (2022)
- [Text Revision by On-the-Fly Representation Optimization](https://mlsanity.com/paper/2204.07359/) (2022)
- [Diamonds in the Rough: Generating Fluent Sentences from Early-Stage Drafts for Academic Writing Assistance](https://mlsanity.com/paper/1910.09180/) (2019)
- [CASIMIR: A Corpus of Scientific Articles enhanced with Multiple Author-Integrated Revisions](https://mlsanity.com/paper/2403.00241/) (2024)
- [Langsmith: An Interactive Academic Text Revision System](https://mlsanity.com/paper/2010.04332/) (2020)
- [How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?](https://mlsanity.com/paper/2402.10770/) (2024)
- [mEdIT: Multilingual Text Editing via Instruction Tuning](https://mlsanity.com/paper/2402.16472/) (2024)
- [ParaRev: Building a dataset for Scientific Paragraph Revision annotated with revision instruction](https://mlsanity.com/paper/2501.05222/) (2025)
- [Text revision in Scientific Writing Assistance: An Overview](https://mlsanity.com/paper/2303.16726/) (2023)
- [Fine-Tuning Language Models for Scientific Writing Support](https://mlsanity.com/paper/2306.10974/) (2023)

## Built on by (most cited first, 6 of 6)

- [Expert Preference-based Evaluation of Automated Related Work Generation](https://mlsanity.com/paper/2508.07955/) (2025)
- [Reward Modeling for Scientific Writing Evaluation](https://mlsanity.com/paper/2601.11374/) (2026)
- [*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation](https://mlsanity.com/paper/2602.15778/) (2026)
- [EditPropBench: Measuring Factual Edit Propagation in Scientific Manuscripts](https://mlsanity.com/paper/2605.02083/) (2026)
- [Evaluating the Homogeneity of Keyphrase Prediction Models](https://mlsanity.com/paper/2602.12989/) (2026)
- [Heterogeneous Element-Aware Cross-Version Differencing of Scientific Documents via Layout-Aware Alignment and Structure-Aware Reasoning](https://mlsanity.com/paper/2607.14117/) (2026)

## Abstract (arXiv)

Evaluating text revision in scientific writing remains a challenge, as traditional metrics such as ROUGE and BERTScore primarily focus on similarity rather than capturing meaningful improvements. In this work, we analyse and identify the limitations of these metrics and explore alternative evaluation methods that better align with human judgments. We first conduct a manual annotation study to assess the quality of different revisions. Then, we investigate reference-free evaluation metrics from related NLP domains. Additionally, we examine LLM-as-a-judge approaches, analysing their ability to assess revisions with and without a gold reference. Our results show that LLMs effectively assess instruction-following but struggle with correctness, while domain-specific metrics provide complementary insights. We find that a hybrid approach combining LLM-as-a-judge evaluation and task-specific metrics offers the most reliable assessment of revision quality.