mlsanityData to 7 Oct 2026

Aligning Large Language Models with Representation Editing: A Control Perspective

arXiv 2406.05954 ↗First posted 10 Jun 2024cs.AI cs.LG cs.SYcited by 62

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) with human objectives is crucial for real-world applications. However, fine-tuning LLMs for alignment often suffers from unstable training and requires substantial computing resources. Test-time alignment techniques, such as prompting and guided decoding, do not modify the underlying model, and their performance remains dependent on the original model's capabilities. To address these challenges, we propose aligning LLMs through representation editing. The core of our method is to view a pre-trained autoregressive LLM as a discrete-time stochastic dynamical system. To achieve alignment for specific objectives, we introduce external control signals into the state space of this language dynamical system. We train a value function directly on the hidden states according to the Bellman equation, enabling gradient-based optimization to obtain the optimal control signals at test time. Our experiments demonstrate that our method outperforms existing test-time alignment techniques while requiring significantly fewer resources compared to fine-tuning methods. Our code is available at https://github.com/Lingkai-Kong/RE-Control.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 73 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

51 of 73 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Value Augmented Sampling for Language Model Alignment and Personalization10 May
  2. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  3. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  4. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  5. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  6. 2023Aligning Large Language Models with Human Preferences through Representation Engineering26 Dec
  7. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  8. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  9. 2023Preference Ranking Optimization for Human Alignment30 Jun
  10. 2022Training language models to follow instructions with human feedback4 Mar
  11. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
28 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  2. 2021Ethical and social risks of harm from Language Modelscited by 957
  3. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  7. 2023The Falcon Series of Open Language Modelscited by 526
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  9. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  10. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  11. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  12. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  13. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  14. 2022A Contrastive Framework for Neural Text Generationcited by 224
  15. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  16. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  17. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  18. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  19. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  20. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  21. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  22. 2023Controlled Decoding from Language Modelscited by 133
  23. 2021FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Financecited by 57
  24. 2023What's the Magic Word? A Control Theory of LLM Promptingcited by 41
  25. 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
  26. 2023Taming AI Bots: Controllability of Neural States in Large Language Modelscited by 22
  27. 2023Prompt Engineering Through the Lens of Optimal Controlcited by 16
  28. 2020Visual Navigation Among Humans with Optimal Control as a Supervisorcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 31; 2026: 24.

62 papers.

Sort:

By area:

  1. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  2. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  3. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 47
  4. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  5. 2025The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representationscited by 26
  6. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  7. 2025Activation Steering for Chain-of-Thought Compressioncited by 22
  8. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attackscited by 20
  9. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  10. 2024Mixture of Attentions For Speculative Decodingcited by 17
  11. 2025Personalize Your LLM: Fake it then Align itcited by 15
  12. 2026ODESteer: A Unified ODE-Based Steering Framework for LLM Alignmentcited by 15
38 more
  1. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  2. 2024Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Modelscited by 12
  3. 2025Activation Steering with a Feedback Controllercited by 11
  4. 2024Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generationcited by 10
  5. 2025Preemptive Detection and Steering of LLM Misalignment via Latent Reachabilitycited by 10
  6. 2024Towards Foundation-model-based Multiagent System to Accelerate AI for Social Impactcited by 9
  7. 2025Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editingcited by 9
  8. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  9. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 8
  10. 2026Local Linearity of LLMs Enables Activation Steering via Model-Based Linear Optimal Controlcited by 8
  11. 2025Inference-time Alignment in Continuous Spacecited by 7
  12. 2025Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decodingcited by 7
  13. 2025Pretrain Value, Not Reward: Decoupled Value Policy Optimizationcited by 6
  14. 2025LLM-ML Teaming: Integrated Symbolic Decoding and Gradient Search for Valid and Stable Generative Feature Transformationcited by 5
  15. 2025Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modelingcited by 4
  16. 2026Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Controlcited by 3
  17. 2025Test-Time Alignment for Large Language Models via Textual Model Predictive Controlcited by 2
  18. 2025Compositional Subspace Representation Fine-tuning for Adaptive Large Language Modelscited by 2
  19. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignmentcited by 2
  20. 2025Balancing Stylization and Truth via Disentangled Representation Steeringcited by 2
  21. 2025Precise Attribute Intensity Control in Large Language Models via Targeted Representation Editingcited by 2
  22. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspectivecited by 2
  23. 2026Inference-time Alignment via Sparse Junction Steeringcited by 2
  24. 2026Activation Steering of Video Generation Models via Reduced-Order Linear Optimal Controlcited by 2
  25. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approachcited by 1
  26. 2025Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundariescited by 1
  27. 2025From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrailscited by 1
  28. 2026Prompting Underestimates LLM Capability for Time Series Classificationcited by 1
  29. 2026FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliancecited by 1
  30. 2026Understanding the Theoretical Foundations of Deep Neural Networks through Differential Equationscited by 1
  31. 2026DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignmentcited by 1
  32. 2026Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behaviorcited by 1
  33. 2026Towards Diverse Scientific Hypothesis Search with Large Language Modelscited by 1
  34. 2026Recipes for Steering and Scaling LLMs via Samplingcited by 1
  35. 2025Dual Filter: A Transformer-like Inference Architecture for Hidden Markov Modelscited by 0
  36. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  37. 2025The Idola Tribus of AI: Large Language Models tend to perceive order where none existscited by 0
  38. 2025Beyond Neural Incompatibility: Cross-Scale Knowledge Transfer in Language Models through Latent Semantic Alignmentcited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  3. 2025A Survey on Training-free Alignment of Large Language Models
  4. 2026Alignment Dynamics in LLM Fine-Tuning
  5. 2023Aligning Large Language Models through Synthetic Feedback
  6. 2023Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models
  7. 2024Efficient Alignment of Large Language Models via Data Sampling
  8. 2023Large Language Model Alignment: A Survey