mlsanityData to 14 Oct 2026

PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics

arXiv 2412.16120 ↗First posted 20 Dec 2024cs.CLcited by 4

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Evaluating the quality of machine-generated natural language content is a challenging task in Natural Language Processing (NLP). Recently, large language models (LLMs) like GPT-4 have been employed for this purpose, but they are computationally expensive due to the extensive token usage required by complex evaluation prompts. In this paper, we propose a prompt optimization approach that uses a smaller, fine-tuned language model to compress input data for evaluation prompt, thus reducing token usage and computational cost when using larger LLMs for downstream evaluation. Our method involves a two-stage fine-tuning process: supervised fine-tuning followed by preference optimization to refine the model's outputs based on human preferences. We focus on Machine Translation (MT) evaluation and utilize the GEMBA-MQM metric as a starting point. Our results show a 2.37× reduction in token usage without any loss in evaluation quality. This work makes state-of-the-art LLM-based metrics like GEMBA-MQM more cost-effective and efficient, enhancing their accessibility for broader use.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 40 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

25 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  6. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  7. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  8. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  9. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  10. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  11. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 146
  12. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
12 more
  1. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  2. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 74
  3. 2024PRewrite: Prompt Rewriting with Reinforcement Learningcited by 45
  4. 2020Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMRcited by 25
  5. 2021Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factorscited by 25
  6. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  7. 2021FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generationcited by 19
  8. 2023There's no Data Like Better Data: Using QE Metrics for MT Data Filteringcited by 19
  9. 2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 16
  10. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  11. 2024xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metricscited by 7
  12. 2022EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metricscited by 4

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025Argument Summarization and its Evaluation in the Era of Large Language Modelscited by 4
  2. 2025BatchGEMBA: Token-Efficient Machine Translation Evaluation with Batched Prompting and Prompt Compressioncited by 3
  3. 2025ContrastScore: Towards Higher Quality, Less Biased, More Efficient Evaluation Metrics with Contrastive Evaluationcited by 1
  4. 2025Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressorcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024What do Large Language Models Need for Machine Translation Evaluation?
  2. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
  3. 2025An Empirical Study on Prompt Compression for Large Language Models
  4. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
  5. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
  6. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMs
  7. 2024Prompt Compression for Large Language Models: A Survey
  8. 2023Little Giants: Exploring the Potential of Small LLMs as Evaluation Metrics in Summarization in the Eval4NLP 2023 Shared Task