# Systematic Characterization of the Effectiveness of Alignment in Large Language Models for Categorical Decisions

arXiv: https://arxiv.org/abs/2409.18995 · HTML: https://mlsanity.com/paper/2409.18995/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 1 of its 17 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Medical LLM alignment](https://mlsanity.com/alignment/medical-llm-alignment/)

## Builds on (most cited first, 1 of 1)

- [Incomplete Contracting and AI Alignment](https://mlsanity.com/paper/1804.04268/) (2018)

## Abstract (arXiv)

As large language models (LLMs) are deployed in high-stakes domains like healthcare, understanding how well their decision-making aligns with human preferences and values becomes crucial, especially when we recognize that there is no single gold standard for these preferences. This paper applies a systematic methodology for evaluating preference alignment in LLMs on categorical decision-making with medical triage as a domain-specific use case. It also measures how effectively an alignment procedure will change the alignment of a specific model. Key to this methodology is a novel simple measure, the Alignment Compliance Index (ACI), that quantifies how effectively a LLM can be aligned to a given preference function or gold standard. Since the ACI measures the effect rather than the process of alignment, it is applicable to alignment methods beyond the in-context learning used in this study. Using a dataset of simulated patient pairs, three frontier LLMs (GPT4o, Claude 3.5 Sonnet, and Gemini Advanced) were assessed on their ability to make triage decisions consistent with an expert clinician's preferences. The models' performance before and after alignment attempts was evaluated using various prompting strategies. The results reveal significant variability in alignment effectiveness across models and alignment approaches. Notably, models that performed well, as measured by ACI, pre-alignment sometimes degraded post-alignment, and small changes in the target preference function led to large shifts in model rankings. The implicit ethical principles, as understood by humans, underlying the LLMs' decisions were also explored through targeted questioning. This study motivates the use of a practical set of methods and the ACI, in the near term, to understand the correspondence between the variety of human and LLM decision-making values in categorical decision-making such as triage.