mlsanityData to 7 Oct 2026

ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Data

arXiv 2205.12600 ↗First posted 25 May 2022cs.CL cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Large pretrained language models have been performing increasingly well in a variety of downstream tasks via prompting. However, it remains unclear from where the model learns the task-specific knowledge, especially in a zero-shot setup. In this work, we want to find evidence of the model's task-specific competence from pretraining and are specifically interested in locating a very small subset of pretraining data that directly supports the model in the task. We call such a subset supporting data evidence and propose a novel method ORCA to effectively identify it, by iteratively using gradient information related to the downstream task. This supporting data evidence offers interesting insights about the prompted language models: in the tasks of sentiment analysis and textual entailment, BERT shows a substantial reliance on BookCorpus, the smaller corpus of BERT's two pretraining corpora, as well as on pretraining examples that mask out synonyms to the task verbalizers.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 38 of its 48 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

38 of 48 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2019Language Models as Knowledge Bases?cited by 1,922
26 more
  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  5. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  8. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  9. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  10. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  11. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  12. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  13. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  14. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  15. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  16. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  17. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  18. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  19. 2019Optimizing Data Usage via Differentiable Rewardscited by 48
  20. 2021Revisiting Methods for Finding Influential Examplescited by 39
  21. 2021Combining Feature and Instance Attribution to Detect Artifactscited by 38
  22. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  23. 2021Influence Tuning: Demoting Spurious Correlations via Instance Attribution and Instance-Driven Updatescited by 27
  24. 2019Fine-grained Sentiment Analysis with Faithful Attentioncited by 23
  25. 2020Multi-Stage Influence Functioncited by 16
  26. 2021A Novel Sequential Coreset Method for Gradient Descent Algorithmscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 10; 2024: 10; 2025: 8; 2026: 1.

34 papers.

Sort:

By area:

  1. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  2. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  3. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  4. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  5. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  6. 2023A Survey of Large Language Models Attributioncited by 71
  7. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 62
  8. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 52
  9. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  10. 2023"According to ...": Prompting Language Models Improves Quoting from Pre-Training Datacited by 42
  11. 2023Understanding In-Context Learning via Supportive Pretraining Datacited by 41
  12. 2024Scalable Influence and Fact Tracing for Large Language Model Pretrainingcited by 40
22 more
  1. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 37
  2. 2025OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokenscited by 27
  3. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  4. 2024ALiiCE: Evaluating Positional Fine-grained Citation Generationcited by 15
  5. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14
  6. 2022Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Modelscited by 13
  7. 2024Verifiable by Design: Aligning Language Models to Quote from Pre-Training Datacited by 13
  8. 2025Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Modelscited by 13
  9. 2025Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Indexcited by 13
  10. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  11. 2023Data Similarity is Not Enough to Explain Language Model Performancecited by 9
  12. 2024Groundedness in Retrieval-augmented Long-form Generation: An Empirical Studycited by 7
  13. 2023Gradient-Based Automated Iterative Recovery for Parameter-Efficient Tuningcited by 5
  14. 2024LLM Attributor: Interactive Visual Attribution for LLM Generationcited by 5
  15. 2024NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learningcited by 4
  16. 2024Derivational Morphology Reveals Analogical Generalization in Large Language Modelscited by 4
  17. 2026Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMscited by 4
  18. 2025Can We Edit LLMs for Long-Tail Biomedical Knowledge?cited by 3
  19. 2023Language-Agnostic Bias Detection in Language Models with Bias Probingcited by 2
  20. 2025Data-Efficient Hate Speech Detection via Cross-Lingual Nearest Neighbor Retrieval with Limited Labeled Datacited by 2
  21. 2025Tree of Preferences for Diversified Recommendationcited by 2
  22. 2025Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behaviorcited by 0