mlsanityData to 7 Oct 2026

DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning

arXiv 2405.14899 ↗First posted 22 May 2024cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) allows transformer-based language models that are pre-trained on general text to quickly learn a specific task with a few "task demonstrations" without updating their parameters, significantly boosting their flexibility and generality. ICL possesses many distinct characteristics from conventional machine learning, thereby requiring new approaches to interpret this learning paradigm. Taking the viewpoint of recent works showing that transformers learn in context by formulating an internal optimizer, we propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL. We empirically verify the effectiveness of our approach for demonstration attribution while being computationally efficient. Leveraging the results, we then show how DETAIL can help improve model performance in real-world scenarios through demonstration reordering and curation. Finally, we experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 43 of its 71 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

43 of 71 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  10. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
31 more
  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  4. 2022Transformers learn in-context by gradient descentcited by 714
  5. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  8. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  9. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  10. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  11. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  12. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  13. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  14. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  15. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  16. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  17. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  18. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  19. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 121
  20. 2021Sampling Permutations for Shapley Value Estimationcited by 101
  21. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  22. 2023In-Context Learning through the Bayesian Prismcited by 78
  23. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 71
  24. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 70
  25. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 62
  26. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  27. 2023Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformerscited by 29
  28. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 26
  29. 2024Localized Zeroth-Order Prompt Optimizationcited by 23
  30. 2023Prompt Valuation Based on Shapley Valuescited by 10
  31. 2022Probably Approximate Shapley Fairness with Applications in Machine Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learningcited by 5
  2. 2026Interpretability Can Be Actionablecited by 5
  3. 2025In-Context Probing for Membership Inference in Fine-Tuned Language Modelscited by 3
  4. 2025Broaden your SCOPE! Efficient Multi-turn Conversation Planning for LLMs with Semantic Spacecited by 2
  5. 2025Uncovering Scaling Laws for Large Language Models via Inverse Problemscited by 0