MAWILE: Multi-Axis Workbench for Inspecting LLM Evaluators
Not on the map. Why?
Abstract (arXiv)
Large language model (LLM) judges provide a flexible and scalable method for evaluating model and agent outputs, but their verdicts can be sensitive to incidental changes in the evaluated response, judge instructions, and scoring rubric. Existing systems examine important subsets of these failure modes, but auditing a configured judge requires testing both the judge instrument and the items it evaluates. We introduce MAWILE, a developer-facing workbench for auditing judge sensitivity across four surfaces: the judge prompt, judge rubric, target-system input, and target-system output. Given a user-supplied judge and representative evaluation items, MAWILE constructs and validates controlled perturbations, re-executes the judge, and localizes the resulting sensitivity. Each perturbation declares whether the verdict should remain invariant or change in a specified direction, allowing the same system to measure both robustness to irrelevant variations and sensitivity to meaningful changes. MAWILE audits binary, ordinal, and pairwise judges without requiring gold labels. The code for this tool is available at: github.com/megagonlabs/mawile-judge.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 21 references that are in the map. First posted on arXiv 18 Sep 2026.
Builds on
16 of 21 references.
References · 16
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
- 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
- 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
- 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
- 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
- 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judgecited by 33
- 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
- 2025Search Arena: Analyzing Search-Augmented LLMscited by 30
- 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
4 more
- 2026JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systemscited by 10
- 2026Judge Reliability Harness: Stress Testing the Reliability of LLM Judgescited by 6
- 2026Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judgescited by 3
- 2026EvalSense: A Framework for Domain-Specific LLM (Meta-)Evaluationcited by 1
Built on by
Not cited yet.