mlsanityData to 14 Oct 2026

Token-wise Influential Training Data Retrieval for Large Language Models

arXiv 2405.11724 ↗First posted 20 May 2024cs.CL cs.AI cs.CRcited by 20

Not on the map. Why?

Abstract (arXiv)

Given a Large Language Model (LLM) generation, how can we identify which training data led to this generation? In this paper, we proposed RapidIn, a scalable framework adapting to LLMs for estimating the influence of each training data. The proposed framework consists of two stages: caching and retrieval. First, we compress the gradient vectors by over 200,000x, allowing them to be cached on disk or in GPU/CPU memory. Then, given a generation, RapidIn efficiently traverses the cached gradients to estimate the influence within minutes, achieving over a 6,326x speedup. Moreover, RapidIn supports multi-GPU parallelization to substantially accelerate caching and retrieval. Our empirical result confirms the efficiency and effectiveness of RapidIn.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 31 of its 68 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

31 of 68 references.

References · 31

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  5. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  6. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  7. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  8. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  9. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  10. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  11. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  12. 2020Influence Functions in Deep Learning Are Fragilecited by 252
19 more
  1. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  2. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  3. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  4. 2021Challenges in Detoxifying Language Modelscited by 169
  5. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  6. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  7. 2022If Influence Functions are the Answer, Then What is the Question?cited by 144
  8. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  9. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 96
  10. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 94
  11. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 90
  12. 2019Data Cleansing for Models Trained with SGDcited by 67
  13. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 53
  14. 2021Text Detoxification using Large Pre-trained Neural Modelscited by 52
  15. 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 49
  16. 2019Fast and Accurate Network Embeddings via Very Sparse Random Projectioncited by 29
  17. 2018Billion-scale Network Embedding with Iterative Random Projectioncited by 25
  18. 2022Contrastive Error Attribution for Finetuned Language Modelscited by 12
  19. 2023OPORP: One Permutation + One Random Projectioncited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 5.

20 papers.

Sort:

By area:

  1. 2024Do Influence Functions Work on Large Language Models?cited by 20
  2. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 16
  3. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
  4. 2025GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projectioncited by 8
  5. 2025All You Need is One: Capsule Prompt Tuning with a Single Vectorcited by 8
  6. 2025VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generationcited by 6
  7. 2025Where Did It Go Wrong? Attributing Undesirable LLM Behaviors via Representation Gradient Tracingcited by 4
  8. 2024DMin: Scalable Training Data Influence Estimation for Diffusion Modelscited by 3
  9. 2025BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretrainingcited by 3
  10. 2025TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuningcited by 3
  11. 2025Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Modelscited by 2
  12. 2026Randomized Antipodal Search Done Right for Data Pareto Improvement of LLM Unlearningcited by 2
8 more
  1. 2025Online Gradient Boosting Decision Tree: In-Place Updates for Efficient Adding/Deleting Datacited by 1
  2. 2026On Representation Redundancy in Large-Scale Instruction Tuning Data Selectioncited by 1
  3. 2026Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuationcited by 1
  4. 2025Estimating the Effects of Sample Training Orders for Large Language Models without Retrainingcited by 0
  5. 2025Data Security in Large Language Models: Risks, Defense, and Directionscited by 0
  6. 2025RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearningcited by 0
  7. 2026Select or Project? Evaluating Lower-dimensional Vectors for LLM Training Data Explanationscited by 0
  8. 2026Which Influence Are We Estimating? The Role of Counterfactual Specifications in Data Attributioncited by 0