# Black-Box Forensics for Conversational LLM Agents

arXiv: https://arxiv.org/abs/2606.22698 · HTML: https://mlsanity.com/paper/2606.22698/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 73 references that are in the map.
- First posted on arXiv 21 Jun 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Model Equality Testing: Which Model Is This API Serving?](https://mlsanity.com/paper/2410.20247/) (2024)
- [GPT-who: An Information Density-based Machine-Generated Text Detector](https://mlsanity.com/paper/2310.06202/) (2023)
- [TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification](https://mlsanity.com/paper/2402.12991/) (2024)
- [Technical Report on the Pangram AI-Generated Text Classifier](https://mlsanity.com/paper/2402.14873/) (2024)
- [From Text to Source: Results in Detecting Large Language Model-Generated Content](https://mlsanity.com/paper/2309.13322/) (2023)
- [Hide and Seek: Fingerprinting Large Language Models with Evolutionary Learning](https://mlsanity.com/paper/2408.02871/) (2024)
- [Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applications](https://mlsanity.com/paper/2406.06737/) (2024)
- [Invisible Traces: Using Hybrid Fingerprinting to identify underlying LLMs in GenAI Apps](https://mlsanity.com/paper/2501.18712/) (2025)
- [Has My System Prompt Been Used? Large Language Model Prompt Membership Inference](https://mlsanity.com/paper/2502.09974/) (2025)
- [AI Generated Text Detection Using Instruction Fine-tuned Large Language and Transformer-Based Models](https://mlsanity.com/paper/2507.05157/) (2025)
- [Zero-Shot Attribution for Large Language Models: A Distribution Testing Approach](https://mlsanity.com/paper/2506.20197/) (2025)

## Built on by (most cited first, 2 of 2)

- [Token Counts Are Not Model Lineage: A Frozen-Threshold Holdout Study of Black-Box LLM API Fingerprinting](https://mlsanity.com/paper/2608.29930/) (2026)
- [Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behavior](https://mlsanity.com/paper/2609.28559/) (2026)

## Abstract (arXiv)

As LLM-powered scams proliferate, black-box forensics for conversational LLM agents offers a path to accountability for systems hidden behind anonymous endpoints. Identifying the base model behind a chatbot endpoint (attribution), without model parameter access or knowledge of the hidden system prompt, would let investigators trace AI-enabled scams back to the providers whose models power them. Detecting when two endpoints run the exact same system prompt (fingerprinting), even one novel and unseen, would link individual scams into criminal networks and expose silent API changes. We conduct an empirical investigation of both capabilities. Our attribution classifiers identify the base model behind an agent with 98% accuracy from a few turns of non-adversarial conversation. Attribution of system prompts, while possible, requires retraining on a large amount of data for each prompt; system prompts in the wild are unbounded and ever-changing, making this approach costly. To tackle this more open-ended setting, our cross-encoder fingerprinting method achieves an AUC of 0.768 and an F1 of 0.703 on entirely unseen system prompts, and aggregating 50 interaction conversations from each target agent boosts AUC to 0.943. Conversational agents with unseen system prompts can thus be fingerprinted with robust accuracy from a few turns of ordinary conversation.