mlsanityData to 14 Oct 2026

SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition

arXiv 2608.18303 ↗First posted 18 Aug 2026cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

LLM-as-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity. We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions mined directly from the judge's own error cases; requiring no oracle responses, task-specific rubrics, or fine-tuning. On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist, while remaining fully training-free. Per-criterion vote evidence provides an interpretable audit trail for diagnosing label ambiguity and judge failure modes unavailable from a single holistic output token.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 23 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

17 of 23 references.

References · 17

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  12. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
5 more
  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  2. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  3. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  4. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 37
  5. 2025Beyond Pointwise Scores: Decomposed Criteria-Based Evaluation of LLM Responsescited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Are We on the Right Way to Assessing LLM-as-a-Judge?
  2. 2026Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
  3. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
  4. 2025Improving LLM-as-a-Judge Inference with the Judgment Distribution
  5. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judges
  6. 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
  7. 2026LLM Judge Validation Under Sparse Overlap: From Inference to Design
  8. 2026JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols