mlsanityData to 7 Oct 2026

LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformations

arXiv 2609.32717 ↗First posted 26 Sep 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods also fits: Human values and AI feedback alignment

Abstract (arXiv)

Large Language Model (LLM) alignment is intended to ensure that models remain helpful and safe, but its stability under input distributional shift is not yet fully understood. Prior work shows that aligned models can fail under jailbreak prompts, alternative encodings, and cross-lingual transfer, yet these failures are usually studied as attacks rather than controlled probes of alignment generalization. Moreover, existing evidence is largely grounded in natural language variation already represented during pretraining, leaving unresolved whether alignment generalizes with semantic content or remains tied to superficial surface patterns. In this paper, we study this question using synthetic semantic-preserving transformations that are rule-based and invertible, preserving task-relevant meaning while shifting inputs beyond standard linguistic variation. Across four open-weight and four commercial models, under both fine-tuning and in-context learning, we use these transformations as a probe of alignment generalization and identify an empirical pattern we term Alignment--utility asymmetry: once models can operate effectively on transformed inputs, task utility is often substantially retained while alignment failure increases more sharply. For example, adapted GPT-4.1 mini shows only limited utility degradation under transformation while its harmful rate rises from 13.3 to 74.3; Gemini 3 Flash similarly retains near-original utility while its harmful rate increases from 2.3 to 43.0. Taken together, these results suggest that semantic-preserving distribution shifts can expose a recurring gap in how utility and alignment generalize in current LLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 50 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

29 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Aligning Large Language Models with Human: A Survey24 Jul

Other references · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  8. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  9. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  10. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  11. 2024Scaling and evaluating sparse autoencoderscited by 530
  12. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
16 more
  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  3. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  4. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  5. 2024Best-of-N Jailbreakingcited by 71
  6. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  7. 2024Endless Jailbreaks with Bijection Learningcited by 18
  8. 2025Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expressioncited by 16
  9. 2025Adjacent Words, Divergent Intents: Jailbreaking Large Language Models via Task Concurrencycited by 13
  10. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12
  11. 2025Towards Safeguarding LLM Fine-tuning APIs against Cipher Attackscited by 10
  12. 2025Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMscited by 10
  13. 2025LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMscited by 7
  14. 2025Hiding in Plain Sight: A Steganographic Approach to Stealthy LLM Jailbreakscited by 6
  15. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
  16. 2025VERA: Variational Inference Framework for Jailbreaking Large Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2023Large Language Model Alignment: A Survey
  3. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context
  4. 2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
  5. 2025Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMs
  6. 2025The Realignment Problem: When Right becomes Wrong in LLMs
  7. 2026Are Aligned Large Language Models Still Misaligned?
  8. 2024PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignment