mlsanityData to 7 Oct 2026

Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese

arXiv 2605.01630 ↗First posted 2 May 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Rankings produced by holistic LLM-as-a-judge scoring are sensitive to the bias of the chosen judge model. We show that switching to binary rubric scoring with multi-judge filtering removes this sensitivity: decomposing the judgement matters more than the judge model itself. To support this claim, we introduce Prosa, the first real user multi-turn Brazilian Portuguese chat benchmark: 1,000 WildChat conversations scored by three judges from three model families on 16 models. Under filtered rubric scoring the three judges agree on every one of the 16 ranks, whereas under holistic scoring they agree on only 7 of 16. Additionally, the rubric filtering pipeline increases the average score gap between neighbouring models by 47%, thereby improving Prosa's discriminative power. Evaluating a new model on Prosa costs approximately 2.1 when using Gemini 3 Flash as the judge. We release the benchmark and the filtering code to ensure that future models can be assessed under identical conditions. These artifacts also make our rubric-based scoring method reusable beyond Prosa, supporting other open-ended evaluation settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 23 references that are in the map. First posted on arXiv 2 May 2026.

Builds on

20 of 23 references.

References · 20

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  3. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  7. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  8. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  9. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  10. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  11. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 171
  12. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128
8 more
  1. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  2. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  3. 2023Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Examscited by 25
  4. 2023BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXamscited by 15
  5. 2025Automatic Legal Writing Evaluation of LLMscited by 11
  6. 2024Sabi'a-2: A New Generation of Portuguese Large Language Modelscited by 10
  7. 2025PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguesecited by 5
  8. 2026Tucano 2 Cool: Better Open Source LLMs for Portuguesecited by 3

Built on by

Not cited yet.