# When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraints

arXiv: https://arxiv.org/abs/2604.16916 · HTML: https://mlsanity.com/paper/2604.16916/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 18 Apr 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)

## Builds on (most cited first, 15 of 15)

- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://mlsanity.com/paper/2308.06463/) (2023)
- [ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs](https://mlsanity.com/paper/2402.11753/) (2024)
- [Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation](https://mlsanity.com/paper/2311.03348/) (2023)
- [X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents](https://mlsanity.com/paper/2504.13203/) (2025)
- [Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue](https://mlsanity.com/paper/2402.17262/) (2024)
- [SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types](https://mlsanity.com/paper/2410.21965/) (2024)
- [Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction](https://mlsanity.com/paper/2409.16783/) (2024)
- [A Study on Large Language Models' Limitations in Multiple-Choice Question Answering](https://mlsanity.com/paper/2401.07955/) (2024)
- [RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems](https://mlsanity.com/paper/2409.16727/) (2024)
- [Self-Correcting Large Language Models: Generation vs. Multiple Choice](https://mlsanity.com/paper/2511.09381/) (2025)

## Built on by (most cited first, 2 of 2)

- [Safety Measurements for Fine-tuned LLMs Should be Grounded in Capability](https://mlsanity.com/paper/2606.03648/) (2026)
- [When Counterbalancing Hides the Bias: Access-Conditioned Position Lock in Forced-Choice LLM Evaluation](https://mlsanity.com/paper/2607.10202/) (2026)

## Abstract (arXiv)

We identify and systematically characterize a class of task-structural alignment failures in large language models (LLMs): even when the harmful intent remains unchanged, changing the task presentation and output constraints can substantially alter model safety behavior. Specifically, when a harmful request is reformulated as a forced-choice multiple-choice question (MCQ) in which all options are harmful and no refusal option is provided, some models that refuse the equivalent open-ended query instead select, prefer, or justify a harmful option. We evaluate 14 proprietary and open-source models on a bilingual Chinese-English human-authored dataset covering five harm categories, together with 900 model-generated Chinese adversarial MCQs. On human-authored data, attack success rate (ASR) increases sharply as prompts shift from open-ended queries to explicit forced-choice formats, typically peaking under intermediate levels of choice constraint. Model-generated Chinese MCQs further weaken or eliminate the recovery regime observed on human-authored data, driving ASR close to saturation for multiple models. The observed transfer patterns are consistent with stronger generators producing more difficult or boundary-adjacent MCQs, although other properties of the generated inputs may also contribute. We also find that adding an explicit refusal option or a safety preamble substantially reduces ASR for several high-capability models, often to near-zero levels, although their effectiveness varies across target models. These findings suggest that safety evaluations centered on open-ended generation may underestimate risks in structured deployment settings, and that task structure should be treated as an important and diagnosable dimension of safety evaluation and alignment training.