mlsanityData to 14 Oct 2026

Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systems

arXiv 2503.01470 ↗First posted 3 Mar 2025cs.CY cs.AI cs.CRcited by 4

Not on the map. Why?

Abstract (arXiv)

The external evaluation of AI systems is increasingly recognised as a crucial approach for understanding their potential risks. However, facilitating external evaluation in practice faces significant challenges in balancing evaluators' need for system access with AI developers' privacy and security concerns. Additionally, evaluators have reason to protect their own privacy - for example, in order to maintain the integrity of held-out test sets. We refer to the challenge of ensuring both developers' and evaluators' privacy as one of providing mutual privacy. In this position paper, we argue that (i) addressing this mutual privacy challenge is essential for effective external evaluation of AI systems, and (ii) current methods for facilitating external evaluation inadequately address this challenge, particularly when it comes to preserving evaluators' privacy. In making these arguments, we formalise the mutual privacy problem; examine the privacy and access requirements of both model owners and evaluators; and explore potential solutions to this challenge, including through the application of cryptographic and hardware-based approaches.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 29 of its 59 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

29 of 59 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  11. 2024Frontier Models are Capable of In-context Schemingcited by 287
  12. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
17 more
  1. 2024Stealing Part of a Production Language Modelcited by 159
  2. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
  3. 2023Auditing large language models: a three-layered approachcited by 137
  4. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  5. 2023Foundation Models and Fair Usecited by 116
  6. 2024The rising costs of training frontier AI modelscited by 95
  7. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  8. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 80
  9. 2022Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governancecited by 67
  10. 2025Evolving Deeper LLM Thinkingcited by 58
  11. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 47
  12. 2024Algorithmic progress in language modelscited by 31
  13. 2023AI capabilities can be significantly improved without expensive retrainingcited by 28
  14. 2024Verifiable evaluations of machine learning models using zkSNARKscited by 16
  15. 2024Language model developers should report train-test overlapcited by 16
  16. 2024Trustless Audits without Revealing Data or Modelscited by 15
  17. 2023Increased Compute Efficiency and the Diffusion of AI Capabilitiescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 10
  2. 2025Benchmarking is Broken -- Don't Let AI be its Own Judgecited by 8
  3. 2025Against racing to AGI: Cooperation, deterrence, and catastrophic riskscited by 2
  4. 2026NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claimscited by 0