mlsanityData to 14 Oct 2026

Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation

arXiv 2509.05605 ↗First posted 6 Sep 2025cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) require high quality preference datasets to align with human preferences. However, conventional methods for constructing such datasets face significant challenges: reliance on pre-collected instructions often leads to distribution mismatches with target models, while the need for sampling multiple stochastic responses introduces substantial computational overhead. In this work, we explore a paradigm shift by leveraging inherent regulation of LLMs' representation space for efficient and tailored preference dataset construction, named Icon2. Specifically, it first extracts layer-wise direction vectors to encode sophisticated human preferences and then uses these vectors to filter self-synthesized instructions based on their inherent consistency. During decoding, bidirectional inherent control is applied to steer token representations, enabling the precise generation of response pairs with clear alignment distinctions. Experimental results demonstrate significant improvements in both alignment and efficiency. Llama3-8B and Qwen2-7B achieve an average win rate improvement of 13.89% on AlpacaEval 2.0 and 13.45% on Arena-Hard, while reducing computational costs by up to 48.1%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 59 references that are in the map. First posted on arXiv 6 Sep 2025.

Builds on

37 of 59 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs26 Nov
  2. 2024Self-Boosting Large Language Models with Synthetic Preference Data9 Oct
  3. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing12 Jun
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024Aligner: Efficient Alignment by Learning to Correct4 Feb
  6. 2023Aligning Large Language Models with Human Preferences through Representation Engineering26 Dec
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  8. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  9. 2021Ethical and social risks of harm from Language Modelscited by 966
  10. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
17 more
  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  2. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  3. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  4. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  5. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  6. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  7. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  8. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  9. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 195
  10. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 125
  11. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  12. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 118
  13. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  14. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  15. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 91
  16. 2021Identifiability in inverse reinforcement learningcited by 55
  17. 2024Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasetscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  2. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  3. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection
  4. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs
  5. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  6. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  7. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  8. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey