mlsanityData to 7 Oct 2026

Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets

arXiv 2406.08124 ↗First posted 12 Jun 2024cs.CL cs.AIcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

The success of the reward model in distinguishing between responses with subtle safety differences depends critically on the high-quality preference dataset, which should capture the fine-grained nuances of harmful and harmless responses. This motivates the need to develop a dataset involving preference margins, which accurately quantify how harmless one response is compared to another. In this paper, we take the first step to propose an effective and cost-efficient framework to promote the margin-enhanced preference dataset development. Our framework, Legend, Leverages representation engineering to annotate preference datasets. It constructs the specific direction within the LLM's embedding space that represents safety. By leveraging this safety direction, Legend can then leverage the semantic distances of paired responses along this direction to annotate margins automatically. We experimentally demonstrate our effectiveness in both reward modeling and harmless alignment for LLMs. Legend also stands out for its efficiency, requiring only the inference time rather than additional training. This efficiency allows for easier implementation and scalability, making Legend particularly valuable for practical applications in aligning LLMs with safe conversations.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 42 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

39 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance20 Jan
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
23 more
  1. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  5. 2022Toy Models of Superpositioncited by 928
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  7. 2023Steering Language Models With Activation Engineeringcited by 824
  8. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  9. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  10. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  11. 2024The Platonic Representation Hypothesiscited by 362
  12. 2023Language Models Represent Space and Timecited by 306
  13. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  14. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  15. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  16. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  17. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  18. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  19. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 24
  20. 2024The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?cited by 24
  21. 2024Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Modelscited by 18
  22. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  23. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performancecited by 9

Built on by

Cited by year

Citing papers per year: 2025: 5.

5 papers.

  1. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  2. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  3. 2025Can Large Language Models Understand Internet Buzzwords Through User-Generated Contentcited by 2
  4. 2025Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguitycited by 2
  5. 2025Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?
  2. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
  3. 2024Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuning
  4. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?
  5. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  6. 2026Cat-DPO: Category-Adaptive Safety Alignment
  7. 2024RewardBench: Evaluating Reward Models for Language Modeling
  8. 2025Data-adaptive Safety Rules for Training Reward Models