mlsanityData to 7 Oct 2026

Rethinking Atomic Decomposition for LLM Judges: A Prompt-Controlled Study of Reference-Grounded QA Evaluation

arXiv 2603.28005 ↗First posted 30 Mar 2026cs.CLcited by 5

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

When an LLM judge only has to assign a three-way support label to a candidate answer given a reference, does asking it to decompose the answer into atomic claims help, and at what cost? We compare four single-call designs that share the judge model, the inputs, and the level of instruction detail: candidate-side atomic decomposition, a matched holistic rubric, reference-side decomposition that checks whether each reference claim is covered, and a bidirectional combination. Support labels are constructed from TruthfulQA, ASQA, and QAMPARI references (200 questions and 400 rows per dataset). All four designs are run with Opus-4.6, GPT-4.1, and Gemini Flash Lite; Sonnet-4.6 is added for the candidate-side and holistic designs. Candidate-side decomposition is weak where the label depends on completeness: the holistic rubric is more accurate on ASQA and QAMPARI for every judge while using fewer tokens. Reference-side decomposition is 12.5-21.3 points more accurate than holistic on ASQA for 28-29% more tokens, and stays near the holistic ceiling on QAMPARI at 63-69% more. On TruthfulQA misconceptions, candidate-side decomposition is competitive and significantly better for two judges. On a 60-row single-author subset whose labels are looser than strict reference completeness, candidate-side decomposition edges ahead of holistic for every judge, reversing the construction-label order. What a judge decomposes should follow what the label measures.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 5 of its 21 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

5 of 21 references.

References · 5

  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2024Long-form factuality in large language modelscited by 146
  3. 2025FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMscited by 7
  4. 2026Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluationcited by 3
  5. 2024UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  2. 2026How Sensitive Are Safety Benchmarks to Judge Configuration Choices?cited by 3
  3. 2026PReMISE: Policy Rubrics as Measurement Specifications for LLM Judgescited by 2
  4. 2026VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learningcited by 0
  5. 2026Acmite: Mitigating Gender Bias in LLMs through Concept-Guided Mutual Informationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks
  2. 2026SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition
  3. 2026Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
  4. 2026LLM Judge Validation Under Sparse Overlap: From Inference to Design
  5. 2026Targeted Tests for LLM Reasoning: An Audit-Constrained Protocol
  6. 2026Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets
  7. 2026Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
  8. 2025When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validity