mlsanityData to 14 Oct 2026

The Viscosity of Logic: Phase Transitions and Hysteresis in DPO Alignment

arXiv 2601.17260 ↗First posted 24 Jan 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) is often tuned as if increasing alignment pressure (controlled by β) yields progressively "better" behavior. We instead treat β as a control parameter and densely sweep it for three 7B open-weight families under a fixed DPO recipe. In Mistral, capability is sharply non-monotonic: aggregated logic-probe margins become positive only in a narrow band near β ≈ 10-2 and revert outside it, with boundary points that are seed-sensitive. Across architectures under the same sweep, we observe qualitatively different response modes: sharp reorganization in Mistral, selective changes in Llama, and smooth trade-offs in Qwen. Critically, the DPO preference margin can anticorrelate with reasoning capability (Pearson r=-0.91 for Llama logic), so margin-based selection can prefer capability-impaired models. Training path also matters: exposure to high β induces capability losses that persist even after β is reduced (hysteresis). These findings motivate capability-resolved evaluation across the β landscape rather than reliance on margins or aggregate benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 28 references that are in the map. First posted on arXiv 24 Jan 2026.

Builds on

15 of 28 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 11

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  6. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  8. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  9. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  10. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  11. 2014The Loss Surfaces of Multilayer Networkscited by 280

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025D-STEER - Preference Alignment Techniques Learn to Behave, not to Believe -- Beneath the Surface, DPO as Steering Vector Perturbation in Activation Space
  2. 2026Where Reasoning Breaks: Logic-Aware Path Selection by Controlling Logical Connectives in LLMs Reasoning Chains
  3. 2026Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMs
  4. 2026Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models
  5. 2026Disentangling Optimization Scale from Preference Scale in DPO
  6. 2024As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss
  7. 2026Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3
  8. 2026LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization