mlsanityData to 14 Oct 2026

CLEAR: Error Analysis via LLM-as-a-Judge Made Easy

arXiv 2507.18392 ↗First posted 24 Jul 2025cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

The evaluation of Large Language Models (LLMs) increasingly relies on other LLMs acting as judges. However, current evaluation paradigms typically yield a single score or ranking, answering which model is better but not why. While essential for benchmarking, these top-level scores obscure the specific, actionable reasons behind a model's performance. To bridge this gap, we introduce CLEAR, an interactive, open-source package for LLM-based error analysis. CLEAR first generates per-instance textual feedback, then it creates a set of system-level error issues, and quantifies the prevalence of each identified issue. Our package also provides users with an interactive dashboard that allows for a comprehensive error analysis through aggregate visualizations, applies interactive filters to isolate specific issues or score ranges, and drills down to the individual instances that exemplify a particular behavioral pattern. We demonstrate CLEAR analysis for RAG and Math benchmarks, and showcase its utility through a user case study.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 15 of its 22 references that are in the map. First posted on arXiv 24 Jul 2025.

Builds on

15 of 22 references.

References · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 117
  9. 2020From Arguments to Key Points: Towards Automatic Argument Summarizationcited by 61
  10. 2023DelucionQA: Detecting Hallucinations in Domain-specific Question Answeringcited by 43
  11. 2020Quantitative Argument Summarization and Beyond: Cross-Domain Key Point Analysiscited by 32
  12. 2024Genie: Achieving Human Parity in Content-Grounded Datasets Generationcited by 31
3 more
  1. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 29
  2. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 27
  3. 2023QualEval: Qualitative Evaluation for Model Improvementcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Doctor-RAG: A Failure-Aware Repair Framework for Agentic Retrieval-Augmented Generationcited by 5
  2. 2025AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarioscited by 3
  3. 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agentscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
  2. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
  3. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
  4. 2024Evaluating LLMs at Detecting Errors in LLM Responses
  5. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?
  6. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  7. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction
  8. 2024ToolScan: A Benchmark for Characterizing Errors in Tool-Use LLMs