# Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systems

arXiv: https://arxiv.org/abs/2503.01470 · HTML: https://mlsanity.com/paper/2503.01470/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 59 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Stealing Part of a Production Language Model](https://mlsanity.com/paper/2403.06634/) (2024)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)
- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Foundation Models and Fair Use](https://mlsanity.com/paper/2303.15715/) (2023)
- [The rising costs of training frontier AI models](https://mlsanity.com/paper/2405.21015/) (2024)
- [Investigating Data Contamination for Pre-training Language Models](https://mlsanity.com/paper/2401.06059/) (2024)
- [What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus](https://mlsanity.com/paper/2105.02732/) (2021)
- [Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance](https://mlsanity.com/paper/2206.04737/) (2022)
- [Evolving Deeper LLM Thinking](https://mlsanity.com/paper/2501.09891/) (2025)
- [A Safe Harbor for AI Evaluation and Red Teaming](https://mlsanity.com/paper/2403.04893/) (2024)
- [Algorithmic progress in language models](https://mlsanity.com/paper/2403.05812/) (2024)
- [AI capabilities can be significantly improved without expensive retraining](https://mlsanity.com/paper/2312.07413/) (2023)
- [Verifiable evaluations of machine learning models using zkSNARKs](https://mlsanity.com/paper/2402.02675/) (2024)
- [Language model developers should report train-test overlap](https://mlsanity.com/paper/2410.08385/) (2024)
- [Trustless Audits without Revealing Data or Models](https://mlsanity.com/paper/2404.04500/) (2024)
- [Increased Compute Efficiency and the Diffusion of AI Capabilities](https://mlsanity.com/paper/2311.15377/) (2023)

## Built on by (most cited first, 4 of 4)

- [The Singapore Consensus on Global AI Safety Research Priorities](https://mlsanity.com/paper/2506.20702/) (2025)
- [Benchmarking is Broken -- Don't Let AI be its Own Judge](https://mlsanity.com/paper/2510.07575/) (2025)
- [Against racing to AGI: Cooperation, deterrence, and catastrophic risks](https://mlsanity.com/paper/2507.21839/) (2025)
- [NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claims](https://mlsanity.com/paper/2605.08192/) (2026)

## Abstract (arXiv)

The external evaluation of AI systems is increasingly recognised as a crucial approach for understanding their potential risks. However, facilitating external evaluation in practice faces significant challenges in balancing evaluators' need for system access with AI developers' privacy and security concerns. Additionally, evaluators have reason to protect their own privacy - for example, in order to maintain the integrity of held-out test sets. We refer to the challenge of ensuring both developers' and evaluators' privacy as one of providing mutual privacy. In this position paper, we argue that (i) addressing this mutual privacy challenge is essential for effective external evaluation of AI systems, and (ii) current methods for facilitating external evaluation inadequately address this challenge, particularly when it comes to preserving evaluators' privacy. In making these arguments, we formalise the mutual privacy problem; examine the privacy and access requirements of both model owners and evaluators; and explore potential solutions to this challenge, including through the application of cryptographic and hardware-based approaches.