# Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models

arXiv: https://arxiv.org/abs/2510.12044 · HTML: https://mlsanity.com/paper/2510.12044/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 12 of 12)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration](https://mlsanity.com/paper/2406.15951/) (2024)
- [Aligner: Efficient Alignment by Learning to Correct](https://mlsanity.com/paper/2402.02416/) (2024)
- [How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations](https://mlsanity.com/paper/1909.04925/) (2019)
- [Locating and Editing Factual Associations in Mamba](https://mlsanity.com/paper/2404.03646/) (2024)
- [Analyzing Hierarchical Structure in Vision Models with Sparse Autoencoders](https://mlsanity.com/paper/2505.15970/) (2025)
- [Improving LLM Safety and Helpfulness using SFT and DPO: A Study on OPT-350M](https://mlsanity.com/paper/2509.09055/) (2025)
- [Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformers](https://mlsanity.com/paper/2504.10148/) (2025)
- [Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment](https://mlsanity.com/paper/2508.07750/) (2025)

## Built on by (most cited first, 1 of 1)

- [Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection](https://mlsanity.com/paper/2603.28921/) (2026)

## Abstract (arXiv)

Existing alignment techniques for Large Language Models (LLMs), such as Direct Preference Optimization (DPO), typically treat the model as a monolithic entity, applying uniform optimization pressure across all layers. This approach overlooks the functional specialization within the Transformer architecture, where different layers are known to handle distinct tasks from syntax to abstract reasoning. In this paper, we challenge this one-size-fits-all paradigm by introducing Hierarchical Alignment, a novel method that applies targeted DPO to distinct functional blocks of a model's layers: local (syntax), intermediate (logic), and global (factuality). Through a series of controlled experiments on state-of-the-art models like Llama-3.1-8B and Qwen1.5-7B using LoRA for surgical fine-tuning, our results, evaluated by a powerful LLM-as-Judge, demonstrate significant and predictable improvements. Specifically, aligning the local layers (Local-Align) enhances grammatical fluency. More importantly, aligning the global layers (Global-Align) not only improves factual consistency as hypothesized but also proves to be the most effective strategy for enhancing logical coherence, outperforming all baselines. Critically, all hierarchical strategies successfully avoid the "alignment tax" observed in standard DPO, where gains in fluency come at the cost of degraded logical reasoning. These findings establish a more resource-efficient, controllable, and interpretable path for model alignment, highlighting the immense potential of shifting from monolithic optimization to structure-aware surgical fine-tuning to build more advanced and reliable LLMs.