# Tools for Verifying Neural Models' Training Data

arXiv: https://arxiv.org/abs/2307.00682 · HTML: https://mlsanity.com/paper/2307.00682/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 2 Jul 2023.

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Detecting AI Trojans Using Meta Neural Analysis](https://mlsanity.com/paper/1910.03137/) (2019)
- [On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning](https://mlsanity.com/paper/2110.11891/) (2021)
- [Counterfactual Memorization in Neural Language Models](https://mlsanity.com/paper/2112.12938/) (2021)
- [Emergent and Predictable Memorization in Large Language Models](https://mlsanity.com/paper/2304.11158/) (2023)
- [Proof-of-Learning: Definitions and Practice](https://mlsanity.com/paper/2103.05633/) (2021)
- [Manipulating SGD with Data Ordering Attacks](https://mlsanity.com/paper/2104.09667/) (2021)
- [Planting Undetectable Backdoors in Machine Learning Models](https://mlsanity.com/paper/2204.06974/) (2022)
- [An Overview of Backdoor Attacks Against Deep Neural Networks and Possible Defences](https://mlsanity.com/paper/2111.08429/) (2021)
- [Architectural Backdoors in Neural Networks](https://mlsanity.com/paper/2206.07840/) (2022)
- [What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring](https://mlsanity.com/paper/2303.11341/) (2023)
- ["Adversarial Examples" for Proof-of-Learning](https://mlsanity.com/paper/2108.09454/) (2021)
- [Deconstructing Distributions: A Pointwise Framework of Learning](https://mlsanity.com/paper/2202.09931/) (2022)

## Built on by (most cited first, 22 of 22)

- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [On the Societal Impact of Open Foundation Models](https://mlsanity.com/paper/2403.07918/) (2024)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [Seven Security Challenges in Cross-domain Multi-agent LLM Systems](https://mlsanity.com/paper/2505.23847/) (2025)
- [Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Unlearning Completeness](https://mlsanity.com/paper/2506.06112/) (2025)
- [Verde: Verification via Refereed Delegation for Machine Learning Programs](https://mlsanity.com/paper/2502.19405/) (2025)
- [Trained Without My Consent: Detecting Code Inclusion In Language Models Trained on Code](https://mlsanity.com/paper/2402.09299/) (2024)
- [Optimistic Verifiable Training by Controlling Hardware Nondeterminism](https://mlsanity.com/paper/2403.09603/) (2024)
- [SoK: Blockchain-Based Decentralized AI (DeAI)](https://mlsanity.com/paper/2411.17461/) (2024)
- [Machine Learning Models Have a Supply Chain Problem](https://mlsanity.com/paper/2505.22778/) (2025)
- [Towards Data Governance of Frontier AI Models](https://mlsanity.com/paper/2412.03824/) (2024)
- [PoTS: Proof-of-Training-Steps for Backdoor Detection in Large Language Models](https://mlsanity.com/paper/2510.15106/) (2025)
- [Verifiable Fine-Tuning for LLMs: Zero-Knowledge Training Proofs Bound to Data Provenance and Policy](https://mlsanity.com/paper/2510.16830/) (2025)
- [Contextual Confidence and Generative AI](https://mlsanity.com/paper/2311.01193/) (2023)
- [TAO: Tolerance-Aware Optimistic Verification for Floating-Point Neural Networks](https://mlsanity.com/paper/2510.16028/) (2025)
- [OPEN-1B: A Fully Auditable Training Run](https://mlsanity.com/paper/2609.17380/) (2026)
- [A2-DIDM: Privacy-preserving Accumulator-enabled Auditing for Distributed Identity of DNN Model](https://mlsanity.com/paper/2405.04108/) (2024)
- [OVIG: Optimistic Verification of AI Training Integrity via Gradient Signals](https://mlsanity.com/paper/2606.21045/) (2026)
- [Privacy-Preserving AI Verification via Minimal Information Disclosure](https://mlsanity.com/paper/2608.02774/) (2026)
- [Training Witnesses: Trusting the Training without Trusting the Trainer](https://mlsanity.com/paper/2609.33915/) (2026)
- [Why Backdooring Neural Networks is so Easy?](https://mlsanity.com/paper/2609.36117/) (2026)
- [TARE: Weigh a Never-Poisoned Twin Before Reading Backdoor-Defense Costs](https://mlsanity.com/paper/2610.06994/) (2026)

## Abstract (arXiv)

It is important that consumers and regulators can verify the provenance of large neural models to evaluate their capabilities and risks. We introduce the concept of a "Proof-of-Training-Data": any protocol that allows a model trainer to convince a Verifier of the training data that produced a set of model weights. Such protocols could verify the amount and kind of data and compute used to train the model, including whether it was trained on specific harmful or beneficial data sources. We explore efficient verification strategies for Proof-of-Training-Data that are compatible with most current large-model training procedures. These include a method for the model-trainer to verifiably pre-commit to a random seed used in training, and a method that exploits models' tendency to temporarily overfit to training data in order to detect whether a given data-point was included in training. We show experimentally that our verification procedures can catch a wide variety of attacks, including all known attacks from the Proof-of-Learning literature.