mlsanityData to 7 Oct 2026

Prompt Risk Control: A Rigorous Framework for Responsible Deployment of Large Language Models

arXiv 2311.13628 ↗First posted 22 Nov 2023cs.LG cs.AI cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

The recent explosion in the capabilities of large language models has led to a wave of interest in how best to prompt a model to perform a given task. While it may be tempting to simply choose a prompt based on average performance on a validation set, this can lead to a deployment where unexpectedly poor responses are generated, especially for the worst-off users. To mitigate this prospect, we propose Prompt Risk Control, a lightweight framework for selecting a prompt based on rigorous upper bounds on families of informative risk measures. We offer methods for producing bounds on a diverse set of metrics, including quantities that measure worst-case responses and disparities in generation quality across the population of users. In addition, we extend the underlying statistical bounding techniques to accommodate the possibility of distribution shifts in deployment. Experiments on applications such as open-ended chat, medical question summarization, and code generation highlight how such a framework can foster responsible deployment by reducing the risk of the worst outcomes.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 30 of its 54 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

30 of 54 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2022Red Teaming Language Models with Language Modelscited by 983
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  10. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  12. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 360
18 more
  1. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  2. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  3. 2022Confident Adaptive Language Modelingcited by 297
  4. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  5. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  6. 2023Challenges and Applications of Large Language Modelscited by 251
  7. 2023Conformal Language Modelingcited by 163
  8. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 131
  9. 2019Fairness risk measurescited by 85
  10. 2019PAC Confidence Sets for Deep Neural Networks via Calibrated Predictioncited by 59
  11. 2021Fairness in Cardiac MR Image Analysis: An Investigation of Bias Due to Data Imbalance in Deep Learning Based Segmentationcited by 57
  12. 2022Conformal Prediction Under Feedback Covariate Shift for Biomolecular Designcited by 53
  13. 2021PAC Prediction Sets Under Covariate Shiftcited by 42
  14. 2022Prediction Sets Adaptive to Unknown Covariate Shiftcited by 22
  15. 2022Semantic uncertainty intervals for disentangled latent spacescited by 22
  16. 2022A global analysis of metrics used for measuring performance in natural language processingcited by 17
  17. 2022Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictionscited by 14
  18. 2023Distribution-Free Statistical Dispersion Control for Societal Applicationscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 5.

17 papers.

Sort:

By area:

  1. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  2. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
  3. 2024Certifying Counterfactual Bias in LLMscited by 9
  4. 2025Conformal Tail Risk Control for Large Language Model Alignmentcited by 8
  5. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  6. 2026Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarkingcited by 5
  7. 2024Improving Predictor Reliability with Selective Recalibrationcited by 3
  8. 2025Controlling the Risk of Corrupted Contexts for Language Models via Early-Exitingcited by 3
  9. 2025Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priorscited by 3
  10. 2026Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generationcited by 2
  11. 2026Adversarially Robust Control of Conditional Value-at-Risk via Rockafellar-Uryasev Conformal Inferencecited by 2
  12. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
5 more
  1. 2024Dynamic Information Sub-Selection for Decision Supportcited by 1
  2. 2025Performative Risk Control: Calibrating Models for Reliable Deployment under Performativitycited by 1
  3. 2025QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictionscited by 1
  4. 2026SCP-NL2TL: Selective Conformal Prediction with Semantic Verification for Natural Language to Temporal Logic Specificationscited by 1
  5. 2026Quantifying Behavioral Tails in Black-Box Language Modelscited by 0