# The Pitfalls of "Security by Obscurity" And What They Mean for Transparent AI

arXiv: https://arxiv.org/abs/2501.18669 · HTML: https://mlsanity.com/paper/2501.18669/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 185 references that are in the map.
- First posted on arXiv 30 Jan 2025.

## Builds on (most cited first, 27 of 27)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Red-Teaming the Stable Diffusion Safety Filter](https://mlsanity.com/paper/2210.04610/) (2022)
- [A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle](https://mlsanity.com/paper/1901.10002/) (2019)
- [Stealing Part of a Production Language Model](https://mlsanity.com/paper/2403.06634/) (2024)
- [Fairness And Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, And Mitigation Strategies](https://mlsanity.com/paper/2304.07683/) (2023)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning](https://mlsanity.com/paper/2202.05338/) (2022)
- [A Safe Harbor for AI Evaluation and Red Teaming](https://mlsanity.com/paper/2403.04893/) (2024)
- [BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B](https://mlsanity.com/paper/2311.00117/) (2023)
- [Fantastic Copyrighted Beasts and How (Not) to Generate Them](https://mlsanity.com/paper/2406.14526/) (2024)
- [The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligence](https://mlsanity.com/paper/2403.13784/) (2024)
- [Experiences with Improving the Transparency of AI Models and Services](https://mlsanity.com/paper/1911.08293/) (2019)
- [Synthetic Data: Methods, Use Cases, and Risks](https://mlsanity.com/paper/2303.01230/) (2023)
- [Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities](https://mlsanity.com/paper/2402.07039/) (2024)
- [Towards a Data Privacy-Predictive Performance Trade-off](https://mlsanity.com/paper/2201.05226/) (2022)
- [Differentially Private Post-Processing for Fair Regression](https://mlsanity.com/paper/2405.04034/) (2024)
- [Beyond the black box](https://mlsanity.com/paper/1604.08472/) (2016)
- [The Fundamental Limits of Least-Privilege Learning](https://mlsanity.com/paper/2402.12235/) (2024)

## Built on by (most cited first, 2 of 2)

- [Normative Conflicts and Shallow AI Alignment](https://mlsanity.com/paper/2506.04679/) (2025)
- [Understanding Parents' Desires in Moderating Children's Interactions with GenAI Chatbots through LLM-Generated Probes](https://mlsanity.com/paper/2603.03727/) (2026)

## Abstract (arXiv)

Calls for transparency in AI systems are growing in number and urgency from diverse stakeholders ranging from regulators to researchers to users (with a comparative absence of companies developing AI). Notions of transparency for AI abound, each addressing distinct interests and concerns. In computer security, transparency is likewise regarded as a key concept. The security community has for decades pushed back against so-called security by obscurity -- the idea that hiding how a system works protects it from attack -- against significant pressure from industry and other stakeholders. Over the decades, in a community process that is imperfect and ongoing, security researchers and practitioners have gradually built up some norms and practices around how to balance transparency interests with possible negative side effects. This paper asks: What insights can the AI community take from the security community's experience with transparency? We identify three key themes in the security community's perspective on the benefits of transparency and their approach to balancing transparency against countervailing interests. For each, we investigate parallels and insights relevant to transparency in AI. We then provide a case study discussion on how transparency has shaped the research subfield of anonymization. Finally, shifting our focus from similarities to differences, we highlight key transparency issues where modern AI systems present challenges different from other kinds of security-critical systems, raising interesting open questions for the security and AI communities alike.