# Human-Instruction-Free LLM Self-Alignment with Limited Samples

arXiv: https://arxiv.org/abs/2401.06785 · HTML: https://mlsanity.com/paper/2401.06785/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 29 of its 31 references that are in the map.
- First posted on arXiv 6 Jan 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://mlsanity.com/paper/2305.03047/) (2023)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [Memorizing Transformers](https://mlsanity.com/paper/2203.08913/) (2022)
- [RAIN: Your Language Models Can Align Themselves without Finetuning](https://mlsanity.com/paper/2309.07124/) (2023)
- [Self-Alignment with Instruction Backtranslation](https://mlsanity.com/paper/2308.06259/) (2023)
- [Safety Assessment of Chinese Large Language Models](https://mlsanity.com/paper/2304.10436/) (2023)
- [SALMON: Self-Alignment with Instructable Reward Models](https://mlsanity.com/paper/2310.05910/) (2023)
- [LongForm: Effective Instruction Tuning with Reverse Instructions](https://mlsanity.com/paper/2304.08460/) (2023)
- [Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency](https://mlsanity.com/paper/2309.17382/) (2023)
- [Unsupervised Cross-Task Generalization via Retrieval Augmentation](https://mlsanity.com/paper/2204.07937/) (2022)
- [In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuning](https://mlsanity.com/paper/2308.04275/) (2023)
- [Self-QA: Unsupervised Knowledge Guided Language Model Alignment](https://mlsanity.com/paper/2305.11952/) (2023)

## Built on by (most cited first, 25 of 25)

- [Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing](https://mlsanity.com/paper/2404.12253/) (2024)
- [LLM Unlearning via Loss Adjustment with Only Forget Data](https://mlsanity.com/paper/2410.11143/) (2024)
- [SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data](https://mlsanity.com/paper/2505.20347/) (2025)
- [A Theoretical Understanding of Self-Correction through In-context Alignment](https://mlsanity.com/paper/2405.18634/) (2024)
- [Learning or Self-aligning? Rethinking Instruction Fine-tuning](https://mlsanity.com/paper/2402.18243/) (2024)
- [Measuring and Reducing LLM Hallucination without Gold-Standard Answers](https://mlsanity.com/paper/2402.10412/) (2024)
- [Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learning](https://mlsanity.com/paper/2410.06508/) (2024)
- [Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning](https://mlsanity.com/paper/2410.14208/) (2024)
- [Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models](https://mlsanity.com/paper/2411.08733/) (2024)
- [COMMUNITY-CROSS-INSTRUCT: Unsupervised Instruction Generation for Aligning Large Language Models to Online Communities](https://mlsanity.com/paper/2406.12074/) (2024)
- [AIR: Complex Instruction Generation via Automatic Iterative Refinement](https://mlsanity.com/paper/2502.17787/) (2025)
- [Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs](https://mlsanity.com/paper/2601.08198/) (2026)
- [MATHWELL: Generating Educational Math Word Problems Using Teacher Annotations](https://mlsanity.com/paper/2402.15861/) (2024)
- [SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2512.07175/) (2025)
- [Structural Entropy Guided Agent for Detecting and Repairing Knowledge Deficiencies in LLMs](https://mlsanity.com/paper/2505.07184/) (2025)
- [Is Free Self-Alignment Possible?](https://mlsanity.com/paper/2406.03642/) (2024)
- [ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions](https://mlsanity.com/paper/2406.08842/) (2024)
- [Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization](https://mlsanity.com/paper/2509.23371/) (2025)
- [Robust Preference Alignment via Directional Neighborhood Consensus](https://mlsanity.com/paper/2510.20498/) (2025)
- [In-context Demonstration Matters: On Prompt Optimization for Pseudo-Supervision Refinement](https://mlsanity.com/paper/2410.03124/) (2024)
- [Clustering Algorithms and RAG Enhancing Semi-Supervised Text Classification with Large LLMs](https://mlsanity.com/paper/2411.06175/) (2024)
- [Not All Documents Are What You Need for Extracting Instruction Tuning Data](https://mlsanity.com/paper/2505.12250/) (2025)
- [Self-Foveate: Enhancing Diversity and Difficulty of Synthesized Instructions from Unsupervised Text via Multi-Level Foveation](https://mlsanity.com/paper/2507.23440/) (2025)
- [A Hormetic Approach to the Value-Loading Problem: Preventing the Paperclip Apocalypse?](https://mlsanity.com/paper/2402.07462/) (2024)
- [Toward a Theory of Value in AI Alignment](https://mlsanity.com/paper/2608.10327/) (2026)

## Abstract (arXiv)

Aligning large language models (LLMs) with human values is a vital task for LLM practitioners. Current alignment techniques have several limitations: (1) requiring a large amount of annotated data; (2) demanding heavy human involvement; (3) lacking a systematic mechanism to continuously improve. In this work, we study aligning LLMs to a new domain with limited samples (e.g. < 100). We propose an algorithm that can self-align LLMs iteratively without active human involvement. Unlike existing works, our algorithm relies on neither human-crafted instructions nor labeled rewards, significantly reducing human involvement. In addition, our algorithm can self-improve the alignment continuously. The key idea is to first retrieve high-quality samples related to the target domain and use them as In-context Learning examples to generate more samples. Then we use the self-generated samples to finetune the LLM iteratively. We show that our method can unlock the LLMs' self-generalization ability to perform alignment with near-zero human supervision. We test our algorithm on three benchmarks in safety, truthfulness, and instruction-following, and show good performance in alignment, domain adaptability, and scalability.