PromptShift-CRC: Drift-Aware Conformal Risk Control for Foundation Models Under Prompt and Domain Shift
Not on the map. Why?
Abstract (arXiv)
Foundation models are now used in settings where the prompts they receive can change quickly. Users change, topics change, policies change, and the model may suddenly face a kind of request that was rare in the calibration data. This makes fixed calibration risky. Conformal prediction and conformal risk control give model-agnostic ways to control error, but they work best when the calibration data still look like the future data. This paper develops PromptShift CRC, a drift-aware conformal risk control method for foundation-model outputs under prompt and domain shift. The method embeds prompts and responses, measures how far the current prompt stream has moved from the calibration pool, gives more weight to relevant or recent calibration examples, and updates the risk level online after observed violations. It reports three practical diagnostics: realized risk error, prompt drift, and effective calibration size. We give conditions under which the method controls risk up to terms for distribution mismatch and weighted quantile uncertainty. In a synthetic prompt-shift benchmark, static conformal risk control fails sharply after drift, while PromptShift-CRC gives the best coverage among the adaptive baselines considered. We then evaluate the same calibration layer on public benchmark derived streams for question answering, toxicity, summarization factuality, and long-context hallucination risk
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 18 of its 21 references that are in the map. First posted on arXiv 14 Jun 2026.
Builds on
18 of 21 references.
References · 18
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
- 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
- 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
- 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
- 2019Conformal Prediction Under Covariate Shiftcited by 511
- 2021Adaptive Conformal Inference Under Distribution Shiftcited by 360
- 2022Conformal Risk Controlcited by 290
- 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 131
- 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 89
- 2024LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Modelscited by 7
6 more
- 2025Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Modelscited by 6
- 2025COPU: Conformal Prediction for Uncertainty Quantification in Natural Language Generationcited by 5
- 2024Uncertainty Awareness of Large Language Models Under Code Distribution Shifts: A Benchmark Studycited by 3
- 2025When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMscited by 2
- 2026Quantifying non deterministic drift in large language modelscited by 1
- 2026Measuring Distribution Shift in User Prompts and Its Effects on LLM Performancecited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.