SCOPE: Sequential Conformal Probing for Reliable OOD Rejection in LLM Services
Not on the map. Why?
Abstract (arXiv)
Rejecting inputs outside the defined in-distribution (IND) service scope is critical for large language model (LLM) services, where unsupported requests should be filtered before full generation. Existing out-of-distribution (OOD) detectors often rely on final outputs or final-layer representations, leaving unclear where service-boundary signals are most clearly encoded inside the model; they also lack a theoretical guarantee for held-out inputs. In this paper, we introduce SCOPE (Sequential Conformal OOD Probing and Evaluation), a framework that selects a readable hidden layer, constructs a conformal gate with IND calibration, and uses a supermartingale e-process to certify persistent service-boundary evidence. Experiments across multiple LLM backbones and six carefully designed boundary conditions show that SCOPE improves gate-level rejection over standard final-layer detectors, while revealing how different OOD boundaries take different geometric forms in hidden space.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 47 references that are in the map. First posted on arXiv 19 Jun 2026.
Builds on
23 of 47 references.
References · 23
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2020Energy-based Out-of-distribution Detectioncited by 1,299
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
- 2016Distribution-Free Predictive Inference For Regressioncited by 668
- 2023The Falcon Series of Open Language Modelscited by 535
- 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
- 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
11 more
- 2024Distinguishing the Knowable from the Unknowable with Language Modelscited by 47
- 2024On Large Language Models' Hallucination with Regard to Known Factscited by 43
- 2024Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanationscited by 30
- 2025Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Taskscited by 18
- 2021Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurancecited by 16
- 2022Beyond Neyman-Pearson: e-values enable hypothesis testing with a data-driven alphacited by 14
- 2021Detecting Operational Adversarial Examples for Reliable Deep Learningcited by 8
- 2024Enhanced Language Model Truthfulness with Learnable Intervention and Uncertainty Expressioncited by 6
- 2025Polysemantic Dropout: Conformal OOD Detection for Specialized LLMscited by 2
- 2025Linear socio-demographic representations emerge in Large Language Models from indirect cuescited by 2
- 2025LLMPrism: Black-box Performance Diagnosis for Production LLM Training Platformscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.