mlsanityData to 7 Oct 2026

Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs

arXiv 2401.14640 ↗First posted 26 Jan 2024cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Attributed Question Answering (AQA) has attracted wide attention, but there are still several limitations in evaluating the attributions, including lacking fine-grained attribution categories, relying on manual annotations, and failing to compare attributions with only subtle differences. To bridge these gaps, we introduce Complex Attributed Question Answering (CAQA), a large-scale benchmark containing comprehensive attribution categories, automatically generated using Knowledge Graphs (KGs), and complex attribution scenarios. We have conducted extensive experiments to verify the effectiveness of CAQA, including the benchmarking of 25 automatic evaluators, their comparison with human evaluators, the testing of LLM evaluators fine-tuned by CAQA and so on. These experiments also lead to a series of important findings that can benefit the future research of AQA. All the codes and data are publicly accessible at https://github.com/HuuuNan/CAQA-Benchmark.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 25 of its 52 references that are in the map. First posted on arXiv 26 Jan 2024.

Builds on

25 of 52 references.

References · 25

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  6. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  7. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  8. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  9. 2020Knowledge Graphscited by 619
  10. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  11. 2023Augmented Language Models: a Surveycited by 398
  12. 2023Evaluating Verifiability in Generative Search Enginescited by 297
13 more
  1. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  2. 2021Internet-Augmented Dialogue Generationcited by 217
  3. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  4. 2020Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Basescited by 169
  5. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  6. 2023A Survey of Large Language Models Attributioncited by 71
  7. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 55
  8. 2022QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphscited by 42
  9. 2023Neural Graph Reasoning: Complex Logical Query Answering Meets Graph Databasescited by 35
  10. 2023Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphscited by 24
  11. 2023Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attributioncited by 24
  12. 2023Evaluating and Modeling Attribution for Cross-Lingual Question Answeringcited by 21
  13. 2022Knowledge Graph Question Answering Datasets and Their Generalizability: Are They Enough for Future Research?cited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 2; 2026: 6.

16 papers.

Sort:

By area:

  1. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  2. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  3. 2024Post-Hoc Answer Attribution for Grounded and Trustworthy Long Document Comprehension: Task, Insights, and Challengescited by 5
  4. 2026Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuningcited by 5
  5. 2026Large Language Model for OWL Proofscited by 3
  6. 2024On the Capacity of Citation Generation by Large Language Modelscited by 2
  7. 2024A Claim Decomposition Benchmark for Long-form Answer Verificationcited by 2
  8. 2025Decomposing and Revising What Language Models Generatecited by 2
  9. 2024Evaluation of Attribution Bias in Generator-Aware Retrieval-Augmented Large Language Modelscited by 1
  10. 2026Beyond Factual Accuracy: Evaluating Global Reasoning Integrity in RAG Systems with LogicScorecited by 1
  11. 2024On the Influence of Shape, Texture and Color for Learning Semantic Segmentationcited by 0
  12. 2024Contrast All the Time: Learning Time Series Representation from Temporal Consistencycited by 0
4 more
  1. 2025Citation Failure: Definition, Analysis and Efficient Mitigationcited by 0
  2. 2026ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attributioncited by 0
  3. 2026Knowledge Dependency Estimation for Reliable Question Answeringcited by 0
  4. 2026Constructing Evaluation Datasets for Procedural Reasoning: Balancing Naturalness, Grounding, and Multi-Hop Coveragecited by 0