ALMA: Alignment with Minimal Annotation
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Recent approaches to large language model (LLM) alignment typically require millions of human annotations or rely on external aligned models for synthetic data generation. This paper introduces ALMA: Alignment with Minimal Annotation, demonstrating that effective alignment can be achieved using only 9,000 labeled examples -- less than 1% of conventional approaches. ALMA generates large amounts of high-quality synthetic alignment data through new techniques: diverse prompt synthesis via few-shot learning, diverse response generation with multiple model checkpoints, and judge (reward model) enhancement through score aggregation and self-distillation. Using only a pretrained Llama3 base model, 5,000 SFT examples, and 4,000 judge annotations, ALMA achieves performance close to Llama3-Instruct across diverse alignment benchmarks (e.g., 0.1% difference on AlpacaEval 2.0 score). These results are achieved with a multi-round, self-bootstrapped data synthesis and training recipe that continues to improve for 10 rounds, surpassing the typical 3-round ceiling of previous methods. These results suggest that base models already possess sufficient knowledge for effective alignment, and that synthetic data generation methods can expose it.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 28 references that are in the map. First posted on arXiv 5 Dec 2024.
Builds on
18 of 28 references.
Closest ancestors · 4
Same direction, newest first
- 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing12 Jun
- 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
- 2022Training language models to follow instructions with human feedback4 Mar
Other references · 14
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2024Self-Rewarding Language Modelscited by 654
- 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
- 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
- 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
- 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
- 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
- 2024Nemotron-4 340B Technical Reportcited by 143
2 more
Built on by
Citing papers per year: 2025: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Efficient Alignment of Large Language Models via Data Sampling
- 2024Human-Instruction-Free LLM Self-Alignment with Limited Samples
- 2024CodecLM: Aligning Language Models with Tailored Synthetic Data
- 2024Towards Scalable Automated Alignment of LLMs: A Survey
- 2024Aligner: Efficient Alignment by Learning to Correct
- 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
- 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
- 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale