# Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs

arXiv: https://arxiv.org/abs/2401.14640 · HTML: https://mlsanity.com/paper/2401.14640/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 52 references that are in the map.
- First posted on arXiv 26 Jan 2024.

## Builds on (most cited first, 25 of 25)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [Internet-Augmented Dialogue Generation](https://mlsanity.com/paper/2107.07566/) (2021)
- [On Hallucination and Predictive Uncertainty in Conditional Language Generation](https://mlsanity.com/paper/2103.15025/) (2021)
- [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](https://mlsanity.com/paper/2011.07743/) (2020)
- [On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation](https://mlsanity.com/paper/2005.03642/) (2020)
- [A Survey of Large Language Models Attribution](https://mlsanity.com/paper/2311.03731/) (2023)
- [HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attribution](https://mlsanity.com/paper/2307.16883/) (2023)
- [QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs](https://mlsanity.com/paper/2205.12665/) (2022)
- [Neural Graph Reasoning: Complex Logical Query Answering Meets Graph Databases](https://mlsanity.com/paper/2303.14617/) (2023)
- [Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphs](https://mlsanity.com/paper/2307.07312/) (2023)
- [Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution](https://mlsanity.com/paper/2310.05634/) (2023)
- [Evaluating and Modeling Attribution for Cross-Lingual Question Answering](https://mlsanity.com/paper/2305.14332/) (2023)
- [Knowledge Graph Question Answering Datasets and Their Generalizability: Are They Enough for Future Research?](https://mlsanity.com/paper/2205.06573/) (2022)

## Built on by (most cited first, 17 of 17)

- [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://mlsanity.com/paper/2407.01370/) (2024)
- [Logical Consistency of Large Language Models in Fact-checking](https://mlsanity.com/paper/2412.16100/) (2024)
- [Post-Hoc Answer Attribution for Grounded and Trustworthy Long Document Comprehension: Task, Insights, and Challenges](https://mlsanity.com/paper/2406.06938/) (2024)
- [Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning](https://mlsanity.com/paper/2605.05676/) (2026)
- [Large Language Model for OWL Proofs](https://mlsanity.com/paper/2601.12444/) (2026)
- [On the Capacity of Citation Generation by Large Language Models](https://mlsanity.com/paper/2410.11217/) (2024)
- [A Claim Decomposition Benchmark for Long-form Answer Verification](https://mlsanity.com/paper/2410.12558/) (2024)
- [Decomposing and Revising What Language Models Generate](https://mlsanity.com/paper/2509.00765/) (2025)
- [Evaluation of Attribution Bias in Generator-Aware Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2410.12380/) (2024)
- [Beyond Factual Accuracy: Evaluating Global Reasoning Integrity in RAG Systems with LogicScore](https://mlsanity.com/paper/2601.15050/) (2026)
- [On the Influence of Shape, Texture and Color for Learning Semantic Segmentation](https://mlsanity.com/paper/2410.14878/) (2024)
- [Contrast All the Time: Learning Time Series Representation from Temporal Consistency](https://mlsanity.com/paper/2410.15416/) (2024)
- [Citation Failure: Definition, Analysis and Efficient Mitigation](https://mlsanity.com/paper/2510.20303/) (2025)
- [ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attribution](https://mlsanity.com/paper/2602.15769/) (2026)
- [Knowledge Dependency Estimation for Reliable Question Answering](https://mlsanity.com/paper/2605.28047/) (2026)
- [Constructing Evaluation Datasets for Procedural Reasoning: Balancing Naturalness, Grounding, and Multi-Hop Coverage](https://mlsanity.com/paper/2606.12767/) (2026)
- [Is the Future Predictable? Modeling Future Predictability for Reliable Temporal Reasoning](https://mlsanity.com/paper/2610.14073/) (2026)

## Abstract (arXiv)

Attributed Question Answering (AQA) has attracted wide attention, but there are still several limitations in evaluating the attributions, including lacking fine-grained attribution categories, relying on manual annotations, and failing to compare attributions with only subtle differences. To bridge these gaps, we introduce Complex Attributed Question Answering (CAQA), a large-scale benchmark containing comprehensive attribution categories, automatically generated using Knowledge Graphs (KGs), and complex attribution scenarios. We have conducted extensive experiments to verify the effectiveness of CAQA, including the benchmarking of 25 automatic evaluators, their comparison with human evaluators, the testing of LLM evaluators fine-tuned by CAQA and so on. These experiments also lead to a series of important findings that can benefit the future research of AQA. All the codes and data are publicly accessible at https://github.com/HuuuNan/CAQA-Benchmark.