# AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution

arXiv: https://arxiv.org/abs/2411.15102 · HTML: https://mlsanity.com/paper/2411.15102/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 63 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://mlsanity.com/paper/2210.08726/) (2022)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [A Survey on Data Selection for Language Models](https://mlsanity.com/paper/2402.16827/) (2024)
- [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [Visualizing and Understanding Neural Models in NLP](https://mlsanity.com/paper/1506.01066/) (2015)

## Built on by (most cited first, 23 of 23)

- [CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attribution](https://mlsanity.com/paper/2602.07918/) (2026)
- [Learning to Attribute with Attention](https://mlsanity.com/paper/2504.13752/) (2025)
- [Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.16415/) (2025)
- [Who Taught the Lie? Responsibility Attribution for Poisoned Knowledge in Retrieval-Augmented Generation](https://mlsanity.com/paper/2509.13772/) (2025)
- [The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution](https://mlsanity.com/paper/2601.15075/) (2026)
- [Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMs](https://mlsanity.com/paper/2602.01914/) (2026)
- [MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering](https://mlsanity.com/paper/2505.24040/) (2025)
- [Large Language Model Sourcing: A Survey](https://mlsanity.com/paper/2510.10161/) (2025)
- [MaxShapley: Towards Incentive-compatible Generative Search with Fair Context Attribution](https://mlsanity.com/paper/2512.05958/) (2025)
- [Explaining the Reasoning of Large Language Models Using Attribution Graphs](https://mlsanity.com/paper/2512.15663/) (2025)
- [Know Or Not: a library for evaluating out-of-knowledge base robustness](https://mlsanity.com/paper/2505.13545/) (2025)
- [Style Attack Disguise: When Fonts Become a Camouflage for Adversarial Intent](https://mlsanity.com/paper/2510.19641/) (2025)
- [Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks](https://mlsanity.com/paper/2604.17761/) (2026)
- [In-Context Credit Assignment via the Core](https://mlsanity.com/paper/2605.06920/) (2026)
- [RDAR: Reward-Driven Agent Relevance Estimation for Autonomous Driving](https://mlsanity.com/paper/2509.19789/) (2025)
- [RREDCoT: Segment-Level Reward Redistribution for Reasoning Models](https://mlsanity.com/paper/2606.06475/) (2026)
- [Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP)](https://mlsanity.com/paper/2607.04223/) (2026)
- [RoPE-LIME: RoPE-Space Locality + Sparse-K Sampling for Efficient LLM Attribution](https://mlsanity.com/paper/2602.06275/) (2026)
- [TRUE: A Trustworthy Unified Explanation Framework for Large Language Model Reasoning](https://mlsanity.com/paper/2602.18905/) (2026)
- [Knowledge Dependency Estimation for Reliable Question Answering](https://mlsanity.com/paper/2605.28047/) (2026)
- [Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?](https://mlsanity.com/paper/2606.32008/) (2026)
- [How Context Attribution Handles What the Model Already Knows](https://mlsanity.com/paper/2607.23804/) (2026)
- [Auditing Agent Actions through Query-Conditioned Attribution](https://mlsanity.com/paper/2609.33676/) (2026)

## Abstract (arXiv)

The influence of contextual input on the behavior of large language models (LLMs) has prompted the development of context attribution methods that aim to quantify each context span's effect on an LLM's generations. The leave-one-out (LOO) error, which measures the change in the likelihood of the LLM's response when a given span of the context is removed, provides a principled way to perform context attribution, but can be prohibitively expensive to compute for large models. In this work, we introduce AttriBoT, a series of novel techniques for efficiently computing an approximation of the LOO error for context attribution. Specifically, AttriBoT uses cached activations to avoid redundant operations, performs hierarchical attribution to reduce computation, and emulates the behavior of large target models with smaller proxy models. Taken together, AttriBoT can provide a >300x speedup while remaining more faithful to a target model's LOO error than prior context attribution methods. This stark increase in performance makes computing context attributions for a given response 30x faster than generating the response itself, empowering real-world applications that require computing attributions at scale. We release a user-friendly and efficient implementation of AttriBoT to enable efficient LLM interpretability as well as encourage future development of efficient context attribution methods.