# CALM: Curiosity-Driven Auditing for Large Language Models

arXiv: https://arxiv.org/abs/2501.02997 · HTML: https://mlsanity.com/paper/2501.02997/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 6 Jan 2025.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Provably Efficient Maximum Entropy Exploration](https://mlsanity.com/paper/1812.02690/) (2018)
- [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024)
- [Automatically Auditing Large Language Models via Discrete Optimization](https://mlsanity.com/paper/2303.04381/) (2023)
- [LM vs LM: Detecting Factual Errors via Cross Examination](https://mlsanity.com/paper/2305.13281/) (2023)
- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Curiosity-driven Red-teaming for Large Language Models](https://mlsanity.com/paper/2402.19464/) (2024)
- [A Survey on Robotics with Foundation Models: toward Embodied AI](https://mlsanity.com/paper/2402.02385/) (2024)
- [Adversarially Guided Actor-Critic](https://mlsanity.com/paper/2102.04376/) (2021)
- [MADE: Exploration via Maximizing Deviation from Explored Regions](https://mlsanity.com/paper/2106.10268/) (2021)
- [Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies](https://mlsanity.com/paper/2109.06974/) (2021)
- [Constrained Intrinsic Motivation for Reinforcement Learning](https://mlsanity.com/paper/2407.09247/) (2024)
- [Toward Evaluating Robustness of Reinforcement Learning with Adversarial Policy](https://mlsanity.com/paper/2305.02605/) (2023)

## Built on by (most cited first, 6 of 6)

- [A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents](https://mlsanity.com/paper/2506.23844/) (2025)
- [CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIs](https://mlsanity.com/paper/2505.13778/) (2025)
- [Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models](https://mlsanity.com/paper/2507.02870/) (2025)
- [RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teaming](https://mlsanity.com/paper/2506.04302/) (2025)
- [RedTopic: Toward Topic-Diverse Red Teaming of Large Language Models](https://mlsanity.com/paper/2507.00026/) (2025)
- [OptiLeak: Efficient Prompt Reconstruction via Reinforcement Learning in Multi-tenant LLM Services](https://mlsanity.com/paper/2602.20595/) (2026)

## Abstract (arXiv)

Auditing Large Language Models (LLMs) is a crucial and challenging task. In this study, we focus on auditing black-box LLMs without access to their parameters, only to the provided service. We treat this type of auditing as a black-box optimization problem where the goal is to automatically uncover input-output pairs of the target LLMs that exhibit illegal, immoral, or unsafe behaviors. For instance, we may seek a non-toxic input that the target LLM responds to with a toxic output or an input that induces the hallucinative response from the target LLM containing politically sensitive individuals. This black-box optimization is challenging due to the scarcity of feasible points, the discrete nature of the prompt space, and the large search space. To address these challenges, we propose Curiosity-Driven Auditing for Large Language Models (CALM), which uses intrinsically motivated reinforcement learning to finetune an LLM as the auditor agent to uncover potential harmful and biased input-output pairs of the target LLM. CALM successfully identifies derogatory completions involving celebrities and uncovers inputs that elicit specific names under the black-box setting. This work offers a promising direction for auditing black-box LLMs. Our code is available at https://github.com/x-zheng16/CALM.git.