# Aligning Model Properties via Conformal Risk Control

arXiv: https://arxiv.org/abs/2406.18777 · HTML: https://mlsanity.com/paper/2406.18777/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 26 Jun 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Fairness in Machine Learning: A Survey](https://mlsanity.com/paper/2010.04053/) (2020)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Supervising strong learners by amplifying weak experts](https://mlsanity.com/paper/1810.08575/) (2018)
- [Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals](https://mlsanity.com/paper/2210.01790/) (2022)
- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [Six Lectures on Linearized Neural Networks](https://mlsanity.com/paper/2308.13431/) (2023)
- [Monotonic classification: an overview on algorithms, performance measures and data sets](https://mlsanity.com/paper/1811.07155/) (2018)
- [Deontological Ethics By Monotonicity Shape Constraints](https://mlsanity.com/paper/2001.11990/) (2020)
- [A Survey on AI Assurance](https://mlsanity.com/paper/2111.07505/) (2021)
- [Low Degree Testing over the Reals](https://mlsanity.com/paper/2204.08404/) (2022)
- [Distribution-Free Testing of Linear Functions on R^n](https://mlsanity.com/paper/1909.03391/) (2019)

## Built on by (most cited first, 7 of 7)

- [Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models](https://mlsanity.com/paper/2506.00911/) (2025)
- [Conformal Policy Control](https://mlsanity.com/paper/2603.02196/) (2026)
- [When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation](https://mlsanity.com/paper/2606.29054/) (2026)
- [On Aligning Prediction Models with Clinical Experiential Learning: A Prostate Cancer Case Study](https://mlsanity.com/paper/2509.04053/) (2025)
- [Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration](https://mlsanity.com/paper/2605.03820/) (2026)
- [Calibrating Conservatism for Scalable Oversight](https://mlsanity.com/paper/2605.28807/) (2026)
- [TRACE-CRC: Trajectory-Adaptive Conformal Risk Control for Multi-Step Channel State Information Prediction](https://mlsanity.com/paper/2608.27124/) (2026)

## Abstract (arXiv)

AI model alignment is crucial due to inadvertent biases in training data and the underspecified machine learning pipeline, where models with excellent test metrics may not meet end-user requirements. While post-training alignment via human feedback shows promise, these methods are often limited to generative AI settings where humans can interpret and provide feedback on model outputs. In traditional non-generative settings with numerical or categorical outputs, detecting misalignment through single-sample outputs remains challenging, and enforcing alignment during training requires repeating costly training processes. In this paper we consider an alternative strategy. We propose interpreting model alignment through property testing, defining an aligned model f as one belonging to a subset P of functions that exhibit specific desired behaviors. We focus on post-processing a pre-trained model f to better align with P using conformal risk control. Specifically, we develop a general procedure for converting queries for testing a given property P to a collection of loss functions suitable for use in a conformal risk control algorithm. We prove a probabilistic guarantee that the resulting conformal interval around f contains a function approximately satisfying P. We exhibit applications of our methodology on a collection of supervised learning datasets for (shape-constrained) properties such as monotonicity and concavity. The general procedure is flexible and can be applied to a wide range of desired properties. Finally, we prove that pre-trained models will always require alignment techniques even as model sizes or training data increase, as long as the training data contains even small biases.