# Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping

arXiv: https://arxiv.org/abs/2402.07610 · HTML: https://mlsanity.com/paper/2402.07610/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 12 Feb 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-rewarding alignment and self-training](https://mlsanity.com/synth/self-rewarding-alignment-and-self-training/)

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Efficient Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2204.02610/) (2022)
- [Towards Stable Test-Time Adaptation in Dynamic Wild World](https://mlsanity.com/paper/2302.12400/) (2023)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://mlsanity.com/paper/2305.03047/) (2023)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [Self-Alignment with Instruction Backtranslation](https://mlsanity.com/paper/2308.06259/) (2023)
- [Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop](https://mlsanity.com/paper/2311.16822/) (2023)
- [LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay](https://mlsanity.com/paper/2310.14985/) (2023)
- [SALMON: Self-Alignment with Instructable Reward Models](https://mlsanity.com/paper/2310.05910/) (2023)
- [Towards Understanding the Interplay of Generative Artificial Intelligence and the Internet](https://mlsanity.com/paper/2306.06130/) (2023)
- [Fairness-guided Few-shot Prompting for Large Language Models](https://mlsanity.com/paper/2303.13217/) (2023)
- [Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search](https://mlsanity.com/paper/2007.07197/) (2020)
- [Are Large Language Models Really Robust to Word-Level Perturbations?](https://mlsanity.com/paper/2309.11166/) (2023)
- [Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledge](https://mlsanity.com/paper/2311.09731/) (2023)
- [Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages](https://mlsanity.com/paper/2310.07418/) (2023)
- [Efficient Toxic Content Detection by Bootstrapping and Distilling Large Language Models](https://mlsanity.com/paper/2312.08303/) (2023)

## Built on by (most cited first, 14 of 14)

- [A Survey on Self-Evolution of Large Language Models](https://mlsanity.com/paper/2404.14387/) (2024)
- [Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences](https://mlsanity.com/paper/2403.07230/) (2024)
- [Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples](https://mlsanity.com/paper/2502.09650/) (2025)
- [I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm](https://mlsanity.com/paper/2408.08072/) (2024)
- [Safety Reasoning with Guidelines](https://mlsanity.com/paper/2502.04040/) (2025)
- [Lifelong Safety Alignment for Language Models](https://mlsanity.com/paper/2505.20259/) (2025)
- [Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation](https://mlsanity.com/paper/2510.13418/) (2025)
- [Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models](https://mlsanity.com/paper/2411.08733/) (2024)
- [QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning](https://mlsanity.com/paper/2408.10504/) (2024)
- [Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generation](https://mlsanity.com/paper/2408.10668/) (2024)
- [MATHWELL: Generating Educational Math Word Problems Using Teacher Annotations](https://mlsanity.com/paper/2402.15861/) (2024)
- [Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment](https://mlsanity.com/paper/2405.20830/) (2024)
- [SHARP: Synthesizing High-quality Aligned Reasoning Problems for Large Reasoning Models Reinforcement Learning](https://mlsanity.com/paper/2505.14147/) (2025)
- [Not All Documents Are What You Need for Extracting Instruction Tuning Data](https://mlsanity.com/paper/2505.12250/) (2025)

## Abstract (arXiv)

Self-alignment is an effective way to reduce the cost of human annotation while ensuring promising model capability. However, most current methods complete the data collection and training steps in a single round, which may overlook the continuously improving ability of self-aligned models. This gives rise to a key query: What if we do multi-time bootstrapping self-alignment? Does this strategy enhance model performance or lead to rapid degradation? In this paper, our pioneering exploration delves into the impact of bootstrapping self-alignment on large language models. Our findings reveal that bootstrapping self-alignment markedly surpasses the single-round approach, by guaranteeing data diversity from in-context learning. To further exploit the capabilities of bootstrapping, we investigate and adjust the training order of data, which yields improved performance of the model. Drawing on these findings, we propose Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced few-shot ability to boost zero or one-shot performance. Based on easy-to-hard training recipe, we propose SOFT+ which further boost self-alignment's performance. Our experiments demonstrate the efficiency of SOFT (SOFT+) across various classification and generation tasks, highlighting the potential of bootstrapping self-alignment on continually enhancing model alignment performance.