mlsanityData to 7 Oct 2026

Measuring, Localizing, and Ablating Alignment Signatures in LLMs

arXiv 2605.30526 ↗First posted 28 May 2026cs.LG cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligned language models often exhibit a recognizable AI-like style, yet its connection to post-training and internal representations remains poorly understood. In this work, we study whether post-training introduces or amplifies AI-like stylistic regularities and whether these regularities have a localized internal signature. To this end, we compare human text, base-model generations, and aligned-model generations under matched human-source prefixes. Aligned generations show lower human-corpus affinity and higher AI-detection rates than base generations, suggesting that post-training shifts generated text away from human-corpus style and toward detector-visible AI-like text. We then introduce PASTA (Post-training Alignment Signature Targeted Ablation), a training-free method that estimates a post-training alignment signature from aligned-base residual contrasts and ablates the corresponding direction during decoding. Across 11 aligned models and 6 AI detectors, PASTA lowers the detection rate for most aligned models; this effect transfers well across detectors and is not reproduced by random directions. Qualitative analysis suggests that PASTA generations remain relevant and coherent while exhibiting greater stylistic variation. Together, these results show that AI-like stylistic effects of post-training can be measured, localized, and causally tested through activation ablation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 38 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

28 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling2 Jun
  2. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  4. 2018Hierarchical Neural Story Generationcited by 1,182
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  6. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  7. 2023Steering Language Models With Activation Engineeringcited by 824
  8. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  9. 2023Whose Opinions Do Language Models Reflect?cited by 759
  10. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  11. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 401
  12. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
13 more
  1. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  2. 2023Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvaturecited by 278
  3. 2024Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Textcited by 255
  4. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  5. 2024Raidar: geneRative AI Detection viA Rewritingcited by 51
  6. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
  7. 2024Technical Report on the Pangram AI-Generated Text Classifiercited by 32
  8. 2023A Ship of Theseus: Curious Cases of Paraphrasing in LLM-Generated Textscited by 23
  9. 2024Nudging: Inference-time Alignment of LLMs via Guided Decodingcited by 22
  10. 2025Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Textcited by 16
  11. 2025Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectorscited by 10
  12. 2026StoryScope: Investigating idiosyncrasies in AI fictioncited by 9
  13. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Large Language Model Alignment: A Survey
  2. 2026From Context Shift to Stylistic Collapse: Why Training Objectives Matter More Than Scale
  3. 2025TRACEALIGN -- Tracing the Drift: Attributing Alignment Failures to Training-Time Belief Sources in LLMs
  4. 2023Aligning Large Language Models with Human: A Survey
  5. 2024Uncovering Factor Level Preferences to Improve Human-Model Alignment
  6. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  7. 2026Ventriloquist LLMs: Linear Alignment of Late-Stage Representations
  8. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges