# Predicting the Performance of Black-box LLMs through Follow-up Queries

arXiv: https://arxiv.org/abs/2501.01558 · HTML: https://mlsanity.com/paper/2501.01558/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 65 references that are in the map.
- First posted on arXiv 2 Jan 2025.

## Builds on (most cited first, 30 of 47)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks](https://mlsanity.com/paper/2306.17844/) (2023)
- [To Believe or Not to Believe Your LLM](https://mlsanity.com/paper/2406.02543/) (2024)

## Built on by (most cited first, 9 of 9)

- [Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs](https://mlsanity.com/paper/2504.04715/) (2025)
- [No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes](https://mlsanity.com/paper/2509.10625/) (2025)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Estimating LLM Consistency: A User Baseline vs Surrogate Metrics](https://mlsanity.com/paper/2505.23799/) (2025)
- [Analyzing Similarity Metrics for Data Selection for Language Model Pretraining](https://mlsanity.com/paper/2502.02494/) (2025)
- [Evaluating Language Model Reasoning about Confidential Information](https://mlsanity.com/paper/2508.19980/) (2025)
- [Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs](https://mlsanity.com/paper/2506.17630/) (2025)
- [Position: Let's Develop Data Probes to Fundamentally Understand How Data Affects LLM Performance](https://mlsanity.com/paper/2605.18801/) (2026)
- [Referential Security as a New Paradigm for AI Evaluations](https://mlsanity.com/paper/2605.25673/) (2026)

## Abstract (arXiv)

Reliably predicting the behavior of language models -- such as whether their outputs are correct or have been adversarially manipulated -- is a fundamentally challenging task. This is often made even more difficult as frontier language models are offered only through closed-source APIs, providing only black-box access. In this paper, we predict the behavior of black-box language models by asking follow-up questions and taking the probabilities of responses as representations to train reliable predictors. We first demonstrate that training a linear model on these responses reliably and accurately predicts model correctness on question-answering and reasoning benchmarks. Surprisingly, this can textiteven outperform white-box linear predictors that operate over model internals or activations. Furthermore, we demonstrate that these follow-up question responses can reliably distinguish between a clean version of an LLM and one that has been adversarially influenced via a system prompt to answer questions incorrectly or to introduce bugs into generated code. Finally, we show that they can also be used to differentiate between black-box LLMs, enabling the detection of misrepresented models provided through an API. Overall, our work shows promise in monitoring black-box language model behavior, supporting their deployment in larger, autonomous systems.