# Differential Assessment of Black-Box AI Agents

arXiv: https://arxiv.org/abs/2203.13236 · HTML: https://mlsanity.com/paper/2203.13236/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 29 references that are in the map.
- First posted on arXiv 24 Mar 2022.

## Builds on (most cited first, 2 of 2)

- [Efficient, Safe, and Probably Approximately Complete Learning of Action Models](https://mlsanity.com/paper/1705.08961/) (2017)
- [Model-Free Model Reconciliation](https://mlsanity.com/paper/1903.07198/) (2019)

## Built on by (most cited first, 6 of 6)

- [Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selection](https://mlsanity.com/paper/2502.09829/) (2025)
- [Autonomous Capability Assessment of Sequential Decision-Making Systems in Stochastic Settings (Extended Version)](https://mlsanity.com/paper/2306.04806/) (2023)
- [A Domain-Independent Agent Architecture for Adaptive Operation in Evolving Open Worlds](https://mlsanity.com/paper/2306.06272/) (2023)
- [Curiosity-Driven Imagination: Discovering Plan Operators and Learning Associated Policies for Open-World Adaptation](https://mlsanity.com/paper/2503.04931/) (2025)
- [Uncovering Systemic and Environment Errors in Autonomous Systems Using Differential Testing](https://mlsanity.com/paper/2507.03870/) (2025)
- [Breaking Task Impasses Quickly: Adaptive Neuro-Symbolic Learning for Open-World Robotics](https://mlsanity.com/paper/2601.16985/) (2026)

## Abstract (arXiv)

Much of the research on learning symbolic models of AI agents focuses on agents with stationary models. This assumption fails to hold in settings where the agent's capabilities may change as a result of learning, adaptation, or other post-deployment modifications. Efficient assessment of agents in such settings is critical for learning the true capabilities of an AI system and for ensuring its safe usage. In this work, we propose a novel approach to "differentially" assess black-box AI agents that have drifted from their previously known models. As a starting point, we consider the fully observable and deterministic setting. We leverage sparse observations of the drifted agent's current behavior and knowledge of its initial model to generate an active querying policy that selectively queries the agent and computes an updated model of its functionality. Empirical evaluation shows that our approach is much more efficient than re-learning the agent model from scratch. We also show that the cost of differential assessment using our method is proportional to the amount of drift in the agent's functionality.