mlsanityData to 7 Oct 2026

Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework

arXiv 2507.08459 ↗First posted 11 Jul 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

With the widespread application of Large Language Models (LLMs) in various tasks, the mainstream LLM platforms generate massive user-model interactions daily. In order to efficiently analyze the performance of models and diagnose failures in their answers, it is essential to develop an automated framework to systematically categorize and attribute errors. However, existing evaluation models lack error attribution capability. In this work, we establish a comprehensive Misattribution Framework with 6 primary and 15 secondary categories to facilitate in-depth analysis. Based on this framework, we present AttriData, a dataset specifically designed for error attribution, encompassing misattribution, along with the corresponding scores and feedback. We also propose MisAttributionLLM, a fine-tuned model on AttriData, which is the first general-purpose judge model capable of simultaneously generating score, misattribution, and feedback. Extensive experiments and analyses are conducted to confirm the effectiveness and robustness of our proposed method.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

35 of 48 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  12. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
23 more
  1. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  2. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  3. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  4. 2023How Language Model Hallucinations Can Snowballcited by 310
  5. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  6. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  7. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  8. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  9. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  10. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  11. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  12. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 65
  13. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Modelscited by 56
  14. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 49
  15. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  16. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Surveycited by 38
  17. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advancescited by 21
  18. 2024An Evaluation of Estimative Uncertainty in Large Language Modelscited by 13
  19. 2023FaMeSumm: Investigating and Improving Faithfulness of Medical Summarizationcited by 9
  20. 2022Why only Micro-F1? Class Weighting of Measures for Relation Classificationcited by 8
  21. 2023TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMscited by 5
  22. 2024IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluationcited by 5
  23. 2025TestNUC: Enhancing Test-Time Computing Approaches and Scaling through Neighboring Unlabeled Data Consistencycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attributioncited by 25
  2. 2026AI Evaluation Should Require Standardized Item-Level Data Releasescited by 1
  3. 2026Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarkscited by 1