# BatchGEMBA: Token-Efficient Machine Translation Evaluation with Batched Prompting and Prompt Compression

arXiv: https://arxiv.org/abs/2503.02756 · HTML: https://mlsanity.com/paper/2503.02756/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Placed in

- Efficient LLM inference and serving / [Prompt and context compression](https://mlsanity.com/inference/prompt-and-context-compression/)

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [Batch Prompting: Efficient Inference with Large Language Model APIs](https://mlsanity.com/paper/2301.08721/) (2023)
- [BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation](https://mlsanity.com/paper/2210.07626/) (2022)
- [500xCompressor: Generalized Prompt Compression for Large Language Models](https://mlsanity.com/paper/2408.03094/) (2024)
- [MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators](https://mlsanity.com/paper/2409.14335/) (2024)
- [BatchEval: Towards Human-like Text Evaluation](https://mlsanity.com/paper/2401.00437/) (2023)
- [PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation](https://mlsanity.com/paper/2406.18528/) (2024)
- [xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics](https://mlsanity.com/paper/2406.14553/) (2024)
- [EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics](https://mlsanity.com/paper/2209.09593/) (2022)
- [PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics](https://mlsanity.com/paper/2412.16120/) (2024)
- [MQM-Chat: Multidimensional Quality Metrics for Chat Translation](https://mlsanity.com/paper/2408.16390/) (2024)

## Built on by (most cited first, 3 of 3)

- [How to Evaluate Speech Translation with Source-Aware Neural MT Metrics](https://mlsanity.com/paper/2511.03295/) (2025)
- [JP-TL-Bench: Anchored Pairwise LLM Evaluation for Bidirectional Japanese-English Translation](https://mlsanity.com/paper/2601.00223/) (2026)
- [Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation](https://mlsanity.com/paper/2609.35824/) (2026)

## Abstract (arXiv)

Recent advancements in Large Language Model (LLM)-based Natural Language Generation evaluation have largely focused on single-example prompting, resulting in significant token overhead and computational inefficiencies. In this work, we introduce BatchGEMBA-MQM, a framework that integrates batched prompting with the GEMBA-MQM metric for machine translation evaluation. Our approach aggregates multiple translation examples into a single prompt, reducing token usage by 2-4 times (depending on the batch size) relative to single-example prompting. Furthermore, we propose a batching-aware prompt compression model that achieves an additional token reduction of 13-15% on average while also showing ability to help mitigate batching-induced quality degradation. Evaluations across several LLMs (GPT-4o, GPT-4o-mini, Mistral Small, Phi4, and CommandR7B) and varying batch sizes reveal that while batching generally negatively affects quality (but sometimes not substantially), prompt compression does not degrade further, and in some cases, recovers quality loss. For instance, GPT-4o retains over 90% of its baseline performance at a batch size of 4 when compression is applied, compared to a 44.6% drop without compression. We plan to release our code and trained models at https://github.com/NL2G/batchgemba to support future research in this domain.