mlsanityData to 7 Oct 2026

Pref-CTRL: Preference Driven LLM Alignment using Representation Editing

arXiv 2604.23543 ↗First posted 26 Apr 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Test-time alignment methods offer a promising alternative to fine-tuning by steering the outputs of large language models (LLMs) at inference time with lightweight interventions on their internal representations. Recently, a prominent and effective approach, RE-Control (Kong et al., 2024), has proposed leveraging an external value function trained over the LLM's hidden states to guide generation via gradient-based editing. While effective, this method overlooks a key characteristic of alignment tasks, i.e. that they are typically formulated as learning from human preferences between candidate responses. To address this, in this paper we propose a novel preference-based training framework, Pref-CTRL, that uses a multi-objective value function to better reflect the structure of preference data. Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets. Our source code is available at https://github.com/UTS-nlPUG/pref-ctrl.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 36 references that are in the map. First posted on arXiv 26 Apr 2026.

Builds on

22 of 36 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model6 May
  2. 2025Online Preference Alignment for Language Models via Count-based Exploration22 Jan
  3. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment10 Oct
  4. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment12 Aug
  5. 2024Aligning Large Language Models with Representation Editing: A Control Perspective10 Jun
  6. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization26 May
  7. 2024Spectral Editing of Activations for Large Language Model Alignment15 May
  8. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model28 Mar
  9. 2024Aligner: Efficient Alignment by Learning to Correct4 Feb
  10. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  11. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  12. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 10

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  5. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  6. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  7. 2025FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steeringcited by 37
  8. 2024Towards Scalable Automated Alignment of LLMs: A Surveycited by 33
  9. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  10. 2025Learning to Reason from Feedback at Test-Timecited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Inference time LLM alignment in single and multidomain preference spectrum
  2. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  3. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
  4. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  5. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  6. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  7. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  8. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignment