mlsanityData to 7 Oct 2026

Human-Instruction-Free LLM Self-Alignment with Limited Samples

arXiv 2401.06785 ↗First posted 6 Jan 2024cs.CL cs.AIcited by 25

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) with human values is a vital task for LLM practitioners. Current alignment techniques have several limitations: (1) requiring a large amount of annotated data; (2) demanding heavy human involvement; (3) lacking a systematic mechanism to continuously improve. In this work, we study aligning LLMs to a new domain with limited samples (e.g. < 100). We propose an algorithm that can self-align LLMs iteratively without active human involvement. Unlike existing works, our algorithm relies on neither human-crafted instructions nor labeled rewards, significantly reducing human involvement. In addition, our algorithm can self-improve the alignment continuously. The key idea is to first retrieve high-quality samples related to the target domain and use them as In-context Learning examples to generate more samples. Then we use the self-generated samples to finetune the LLM iteratively. We show that our method can unlock the LLMs' self-generalization ability to perform alignment with near-zero human supervision. We test our algorithm on three benchmarks in safety, truthfulness, and instruction-following, and show good performance in alignment, domain adaptability, and scalability.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 31 references that are in the map. First posted on arXiv 6 Jan 2024.

Builds on

29 of 31 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  2. 2023SALMON: Self-Alignment with Instructable Reward Models9 Oct
  3. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep
  4. 2023Self-QA: Unsupervised Knowledge Guided Language Model Alignment19 May
  5. 2023LIMA: Less Is More for Alignment18 May
  6. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  12. 2022Large Language Models Can Self-Improvecited by 656
9 more
  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  2. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  3. 2022Memorizing Transformerscited by 219
  4. 2023Self-Alignment with Instruction Backtranslationcited by 149
  5. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  6. 2023LongForm: Effective Instruction Tuning with Reverse Instructionscited by 47
  7. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  8. 2022Unsupervised Cross-Task Generalization via Retrieval Augmentationcited by 42
  9. 2023In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 8; 2026: 2.

25 papers.

Sort:

By area:

  1. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  2. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  3. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  4. 2024Learning or Self-aligning? Rethinking Instruction Fine-tuningcited by 43
  5. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  6. 2024Measuring and Reducing LLM Hallucination without Gold-Standard Answerscited by 31
  7. 2024Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learningcited by 27
  8. 2024COMMUNITY-CROSS-INSTRUCT: Unsupervised Instruction Generation for Aligning Large Language Models to Online Communitiescited by 12
  9. 2024Montessori-Instruct: Generate Influential Training Data Tailored for Student Learningcited by 12
  10. 2024Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Modelscited by 12
  11. 2025AIR: Complex Instruction Generation via Automatic Iterative Refinementcited by 10
  12. 2026Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMscited by 9
13 more
  1. 2024MATHWELL: Generating Educational Math Word Problems Using Teacher Annotationscited by 7
  2. 2025SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Modelscited by 7
  3. 2024Is Free Self-Alignment Possible?cited by 4
  4. 2024ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictionscited by 4
  5. 2025Structural Entropy Guided Agent for Detecting and Repairing Knowledge Deficiencies in LLMscited by 4
  6. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimizationcited by 3
  7. 2025Robust Preference Alignment via Directional Neighborhood Consensuscited by 2
  8. 2024In-context Demonstration Matters: On Prompt Optimization for Pseudo-Supervision Refinementcited by 1
  9. 2024Clustering Algorithms and RAG Enhancing Semi-Supervised Text Classification with Large LLMscited by 1
  10. 2025Not All Documents Are What You Need for Extracting Instruction Tuning Datacited by 1
  11. 2025Self-Foveate: Enhancing Diversity and Difficulty of Synthesized Instructions from Unsupervised Text via Multi-Level Foveationcited by 1
  12. 2024A Hormetic Approach to the Value-Loading Problem: Preventing the Paperclip Apocalypse?cited by 0
  13. 2026Toward a Theory of Value in AI Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Aligning Large Language Models with Human: A Survey
  2. 2024Aligners: Decoupling LLMs and Alignment
  3. 2025A Survey on Training-free Alignment of Large Language Models
  4. 2024Efficient Alignment of Large Language Models via Data Sampling
  5. 2024Towards Scalable Automated Alignment of LLMs: A Survey
  6. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  7. 2024Your Weak LLM is Secretly a Strong Teacher for Alignment
  8. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges