# PromptShift-CRC: Drift-Aware Conformal Risk Control for Foundation Models Under Prompt and Domain Shift

arXiv: https://arxiv.org/abs/2606.15964 · HTML: https://mlsanity.com/paper/2606.15964/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 14 Jun 2026.

## Builds on (most cited first, 18 of 18)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Conformal Prediction with Large Language Models for Multi-Choice Question Answering](https://mlsanity.com/paper/2305.18404/) (2023)
- [SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency](https://mlsanity.com/paper/2311.01740/) (2023)
- [LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models](https://mlsanity.com/paper/2410.09962/) (2024)
- [COPU: Conformal Prediction for Uncertainty Quantification in Natural Language Generation](https://mlsanity.com/paper/2502.12601/) (2025)
- [Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Models](https://mlsanity.com/paper/2506.05497/) (2025)
- [Uncertainty Awareness of Large Language Models Under Code Distribution Shifts: A Benchmark Study](https://mlsanity.com/paper/2402.05939/) (2024)
- [When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMs](https://mlsanity.com/paper/2506.10095/) (2025)
- [Quantifying non deterministic drift in large language models](https://mlsanity.com/paper/2601.19934/) (2026)
- [Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance](https://mlsanity.com/paper/2604.17650/) (2026)

## Built on by (most cited first, 2 of 2)

- [Signed Evidence Flow: Conflict-Aware and Stability-Calibrated Data Analysis](https://mlsanity.com/paper/2606.21875/) (2026)
- [CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds](https://mlsanity.com/paper/2609.22183/) (2026)

## Abstract (arXiv)

Foundation models are now used in settings where the prompts they receive can change quickly. Users change, topics change, policies change, and the model may suddenly face a kind of request that was rare in the calibration data. This makes fixed calibration risky. Conformal prediction and conformal risk control give model-agnostic ways to control error, but they work best when the calibration data still look like the future data. This paper develops PromptShift CRC, a drift-aware conformal risk control method for foundation-model outputs under prompt and domain shift. The method embeds prompts and responses, measures how far the current prompt stream has moved from the calibration pool, gives more weight to relevant or recent calibration examples, and updates the risk level online after observed violations. It reports three practical diagnostics: realized risk error, prompt drift, and effective calibration size. We give conditions under which the method controls risk up to terms for distribution mismatch and weighted quantile uncertainty. In a synthetic prompt-shift benchmark, static conformal risk control fails sharply after drift, while PromptShift-CRC gives the best coverage among the adaptive baselines considered. We then evaluate the same calibration layer on public benchmark derived streams for question answering, toxicity, summarization factuality, and long-context hallucination risk