# MAWILE: Multi-Axis Workbench for Inspecting LLM Evaluators

arXiv: https://arxiv.org/abs/2609.22599 · HTML: https://mlsanity.com/paper/2609.22599/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 18 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [Evaluating Scoring Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2506.22316/) (2025)
- [LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge](https://mlsanity.com/paper/2506.09443/) (2025)
- [Search Arena: Analyzing Search-Augmented LLMs](https://mlsanity.com/paper/2506.05334/) (2025)
- [Finding Blind Spots in Evaluator LLMs with Interpretable Checklists](https://mlsanity.com/paper/2406.13439/) (2024)
- [reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs](https://mlsanity.com/paper/2503.11751/) (2025)
- [JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems](https://mlsanity.com/paper/2604.23478/) (2026)
- [Judge Reliability Harness: Stress Testing the Reliability of LLM Judges](https://mlsanity.com/paper/2603.05399/) (2026)
- [Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges](https://mlsanity.com/paper/2602.13576/) (2026)
- [EvalSense: A Framework for Domain-Specific LLM (Meta-)Evaluation](https://mlsanity.com/paper/2602.18823/) (2026)

## Abstract (arXiv)

Large language model (LLM) judges provide a flexible and scalable method for evaluating model and agent outputs, but their verdicts can be sensitive to incidental changes in the evaluated response, judge instructions, and scoring rubric. Existing systems examine important subsets of these failure modes, but auditing a configured judge requires testing both the judge instrument and the items it evaluates. We introduce MAWILE, a developer-facing workbench for auditing judge sensitivity across four surfaces: the judge prompt, judge rubric, target-system input, and target-system output. Given a user-supplied judge and representative evaluation items, MAWILE constructs and validates controlled perturbations, re-executes the judge, and localizes the resulting sensitivity. Each perturbation declares whether the verdict should remain invariant or change in a specified direction, allowing the same system to measure both robustness to irrelevant variations and sensitivity to meaningful changes. MAWILE audits binary, ordinal, and pairwise judges without requiring gold labels. The code for this tool is available at: github.com/megagonlabs/mawile-judge.