# LogProber: Disentangling confidence from contamination in LLM responses

arXiv: https://arxiv.org/abs/2408.14352 · HTML: https://mlsanity.com/paper/2408.14352/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Builds on (most cited first, 8 of 8)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [Investigating Data Contamination for Pre-training Language Models](https://mlsanity.com/paper/2401.06059/) (2024)
- [PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks](https://mlsanity.com/paper/2404.04671/) (2024)

## Built on by (most cited first, 3 of 3)

- [LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation](https://mlsanity.com/paper/2412.10424/) (2024)
- [CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting](https://mlsanity.com/paper/2505.18102/) (2025)
- [Is This Collection Worth My LLM's Time? Automatically Measuring Information Potential in Text Corpora](https://mlsanity.com/paper/2502.13691/) (2025)

## Abstract (arXiv)

In machine learning, contamination refers to situations where testing data leak into the training set. The issue is particularly relevant for the evaluation of the performance of Large Language Models (LLMs), which are generally trained on gargantuan, and generally opaque, corpora of text scraped from the world wide web. Developing tools to detect contamination is therefore crucial to be able to fairly and properly track the evolution of the performance of LLMs. To date, only a few recent studies have attempted to address the issue of quantifying and detecting contamination in short text sequences, such as those commonly found in benchmarks. However, these methods have limitations that can sometimes render them impractical. In the present paper, we introduce LogProber, a novel, efficient algorithm that we show to be able to detect contamination in a black box setting that tries to tackle some of these drawbacks by focusing on the familiarity with the question rather than the answer. Here, we explore the properties of the proposed method in comparison with concurrent approaches, identify its advantages and limitations, and illustrate how different forms of contamination can go undetected depending on the design of the detection algorithm.