# SCOPE: Sequential Conformal Probing for Reliable OOD Rejection in LLM Services

arXiv: https://arxiv.org/abs/2606.21255 · HTML: https://mlsanity.com/paper/2606.21255/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 19 Jun 2026.

## Builds on (most cited first, 23 of 23)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Distinguishing the Knowable from the Unknowable with Language Models](https://mlsanity.com/paper/2402.03563/) (2024)
- [On Large Language Models' Hallucination with Regard to Known Facts](https://mlsanity.com/paper/2403.20009/) (2024)
- [Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations](https://mlsanity.com/paper/2402.15062/) (2024)
- [Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Tasks](https://mlsanity.com/paper/2506.00823/) (2025)
- [Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurance](https://mlsanity.com/paper/2112.00646/) (2021)
- [Beyond Neyman-Pearson: e-values enable hypothesis testing with a data-driven alpha](https://mlsanity.com/paper/2205.00901/) (2022)
- [Detecting Operational Adversarial Examples for Reliable Deep Learning](https://mlsanity.com/paper/2104.06015/) (2021)
- [Enhanced Language Model Truthfulness with Learnable Intervention and Uncertainty Expression](https://mlsanity.com/paper/2405.00301/) (2024)
- [Polysemantic Dropout: Conformal OOD Detection for Specialized LLMs](https://mlsanity.com/paper/2509.04655/) (2025)
- [Linear socio-demographic representations emerge in Large Language Models from indirect cues](https://mlsanity.com/paper/2512.10065/) (2025)
- [LLMPrism: Black-box Performance Diagnosis for Production LLM Training Platforms](https://mlsanity.com/paper/2505.00342/) (2025)

## Built on by (most cited first, 2 of 2)

- [Same evidence, different judgments: Evidence noncommutative in vision/speech-text conflicts](https://mlsanity.com/paper/2609.26986/) (2026)
- [Do We Need Complex Topology Control? Distinct-Peer Random Routing Improves Cost-Efficiency in Sparse Multi-Agent Debate](https://mlsanity.com/paper/2609.27150/) (2026)

## Abstract (arXiv)

Rejecting inputs outside the defined in-distribution (IND) service scope is critical for large language model (LLM) services, where unsupported requests should be filtered before full generation. Existing out-of-distribution (OOD) detectors often rely on final outputs or final-layer representations, leaving unclear where service-boundary signals are most clearly encoded inside the model; they also lack a theoretical guarantee for held-out inputs. In this paper, we introduce SCOPE (Sequential Conformal OOD Probing and Evaluation), a framework that selects a readable hidden layer, constructs a conformal gate with IND calibration, and uses a supermartingale e-process to certify persistent service-boundary evidence. Experiments across multiple LLM backbones and six carefully designed boundary conditions show that SCOPE improves gate-level rejection over standard final-layer detectors, while revealing how different OOD boundaries take different geometric forms in hidden space.