# LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformations

arXiv: https://arxiv.org/abs/2609.32717 · HTML: https://mlsanity.com/paper/2609.32717/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 29 of 29)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models](https://mlsanity.com/paper/2310.02949/) (2023)
- [FlipAttack: Jailbreak LLMs via Flipping](https://mlsanity.com/paper/2410.02832/) (2024)
- [Best-of-N Jailbreaking](https://mlsanity.com/paper/2412.03556/) (2024)
- [Aligning Large Language Models with Human: A Survey](https://mlsanity.com/paper/2307.12966/) (2023)
- [AdvPrefix: An Objective for Nuanced LLM Jailbreaks](https://mlsanity.com/paper/2412.10321/) (2024)
- [Endless Jailbreaks with Bijection Learning](https://mlsanity.com/paper/2410.01294/) (2024)
- [Adjacent Words, Divergent Intents: Jailbreaking Large Language Models via Task Concurrency](https://mlsanity.com/paper/2510.21189/) (2025)
- [Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expression](https://mlsanity.com/paper/2505.13527/) (2025)
- [GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs](https://mlsanity.com/paper/2411.14133/) (2024)
- [Towards Safeguarding LLM Fine-tuning APIs against Cipher Attacks](https://mlsanity.com/paper/2508.17158/) (2025)
- [Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs](https://mlsanity.com/paper/2510.02833/) (2025)
- [LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMs](https://mlsanity.com/paper/2505.10838/) (2025)
- [No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms](https://mlsanity.com/paper/2502.19537/) (2025)
- [Hiding in Plain Sight: A Steganographic Approach to Stealthy LLM Jailbreaks](https://mlsanity.com/paper/2505.16765/) (2025)
- [VERA: Variational Inference Framework for Jailbreaking Large Language Models](https://mlsanity.com/paper/2506.22666/) (2025)

## Abstract (arXiv)

Large Language Model (LLM) alignment is intended to ensure that models remain helpful and safe, but its stability under input distributional shift is not yet fully understood. Prior work shows that aligned models can fail under jailbreak prompts, alternative encodings, and cross-lingual transfer, yet these failures are usually studied as attacks rather than controlled probes of alignment generalization. Moreover, existing evidence is largely grounded in natural language variation already represented during pretraining, leaving unresolved whether alignment generalizes with semantic content or remains tied to superficial surface patterns. In this paper, we study this question using synthetic semantic-preserving transformations that are rule-based and invertible, preserving task-relevant meaning while shifting inputs beyond standard linguistic variation. Across four open-weight and four commercial models, under both fine-tuning and in-context learning, we use these transformations as a probe of alignment generalization and identify an empirical pattern we term Alignment--utility asymmetry: once models can operate effectively on transformed inputs, task utility is often substantially retained while alignment failure increases more sharply. For example, adapted GPT-4.1 mini shows only limited utility degradation under transformation while its harmful rate rises from 13.3 to 74.3; Gemini 3 Flash similarly retains near-original utility while its harmful rate increases from 2.3 to 43.0. Taken together, these results suggest that semantic-preserving distribution shifts can expose a recurring gap in how utility and alignment generalize in current LLMs.