mlsanityData to 14 Oct 2026

M-Prometheus: A Suite of Open Multilingual LLM Judges

arXiv 2504.04953 ↗First posted 7 Apr 2025cs.CL cs.AIcited by 38

Not on the map. Why?

Abstract (arXiv)

The use of language models for automatically evaluating long-form text (LLM-as-a-judge) is becoming increasingly common, yet most LLM judges are optimized exclusively for English, with strategies for enhancing their multilingual evaluation capabilities remaining largely unexplored in the current literature. This has created a disparity in the quality of automatic evaluation methods for non-English languages, ultimately hindering the development of models with better multilingual capabilities. To bridge this gap, we introduce M-Prometheus, a suite of open-weight LLM judges ranging from 3B to 14B parameters that can provide both direct assessment and pairwise comparison feedback on multilingual outputs. M-Prometheus models outperform state-of-the-art open LLM judges on multilingual reward benchmarks spanning more than 20 languages, as well as on literary machine translation (MT) evaluation covering 4 language pairs. Furthermore, M-Prometheus models can be leveraged at decoding time to significantly improve generated outputs across all 3 tested languages, showcasing their utility for the development of better multilingual models. Lastly, through extensive ablations, we identify the key factors for obtaining an effective multilingual judge, including backbone model selection and training on synthetic multilingual feedback data instead of translated data. We release our models, training dataset, and code.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 18 of its 37 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

18 of 37 references.

References · 18

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  3. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  4. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  5. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  6. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  7. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  8. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 100
  9. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 97
  10. 2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 74
  11. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  12. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codeccited by 44
6 more
  1. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  2. 2024Codec-SUPERB: An In-Depth Analysis of Sound Codec Modelscited by 26
  3. 2024Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generationcited by 23
  4. 2025UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebookcited by 10
  5. 2024Neural Speech Coding for Real-time Communications using Constant Bitrate Scalar Quantizationcited by 4
  6. 2024Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 23.

38 papers.

Sort:

By area:

  1. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 33
  2. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 23
  3. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  4. 2025LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answeringcited by 10
  5. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 10
  6. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  7. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Modelscited by 8
  8. 2026Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modelingcited by 8
  9. 2025Disambiguation in Conversational Question Answering in the Era of LLMs and Agents: A Surveycited by 6
  10. 2025ToolRM: Towards Agentic Tool-Use Reward Modelingcited by 5
  11. 2026FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judgecited by 5
  12. 2025SEADialogues: A Multilingual Culturally Grounded Multi-turn Dialogue Dataset on Southeast Asian Languagescited by 4
26 more
  1. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptationcited by 4
  2. 2025LLM-as-a-qualitative-judge: automating error analysis in natural language generationcited by 3
  3. 2025Checklist Engineering Empowers Multilingual LLM Judgescited by 3
  4. 2026Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judgecited by 3
  5. 2026Cross-Lingual LLM-Judge Transfer via Evaluation Decompositioncited by 3
  6. 2026Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneckcited by 2
  7. 2025MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Languagecited by 1
  8. 2026Consensus Group Relative Policy Optimization for Text Generationcited by 1
  9. 2026Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judgecited by 1
  10. 2026Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generationcited by 1
  11. 2026How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Modelscited by 1
  12. 2026Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speechcited by 1
  13. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  14. 2025Eye of Judgement: Dissecting the Evaluation of Russian-speaking LLMs with POLLUXcited by 0
  15. 2025Unlocking Latent Discourse Translation in LLMs Through Quality-Aware Decodingcited by 0
  16. 2025Comparing Approaches to Automatic Summarization in Less-Resourced Languagescited by 0
  17. 2026Retell, Reward, Repeat: Reinforcement Learning for Narrative Theory-Informed Story Retellingcited by 0
  18. 2026BoRP: Bootstrapped Regression Probing for Scalable and Human-Aligned LLM Evaluationcited by 0
  19. 2026Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Studycited by 0
  20. 2026Model-Based Quality Assessment for Massively Multilingual Parallel Datacited by 0
  21. 2026LEAF: Growing Trees Without Branching for Speech-Aware Large Language Model Post-Trainingcited by 0
  22. 2026CAT-Translate: Building Compact Open-Source Models for Japanese-English Translationcited by 0
  23. 2026Lower-Resource, Higher Scores: Language Bias in LLM Evaluatorscited by 0
  24. 2026Inter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended Textcited by 0
  25. 2026UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigmscited by 0
  26. 2026Unreliable Narrator Framing as a Cost-effective Mechanism for Satire Explanation of Australian and American Satirical Newscited by 0