mlsanityData to 7 Oct 2026

Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping

arXiv 2402.07610 ↗First posted 12 Feb 2024cs.CL cs.AIcited by 14

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Self-alignment is an effective way to reduce the cost of human annotation while ensuring promising model capability. However, most current methods complete the data collection and training steps in a single round, which may overlook the continuously improving ability of self-aligned models. This gives rise to a key query: What if we do multi-time bootstrapping self-alignment? Does this strategy enhance model performance or lead to rapid degradation? In this paper, our pioneering exploration delves into the impact of bootstrapping self-alignment on large language models. Our findings reveal that bootstrapping self-alignment markedly surpasses the single-round approach, by guaranteeing data diversity from in-context learning. To further exploit the capabilities of bootstrapping, we investigate and adjust the training order of data, which yields improved performance of the model. Drawing on these findings, we propose Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced few-shot ability to boost zero or one-shot performance. Based on easy-to-hard training recipe, we propose SOFT+ which further boost self-alignment's performance. Our experiments demonstrate the efficiency of SOFT (SOFT+) across various classification and generation tasks, highlighting the potential of bootstrapping self-alignment on continually enhancing model alignment performance.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 27 of its 41 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

27 of 41 references.

References · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  11. 2022Large Language Models Can Self-Improvecited by 656
  12. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
15 more
  1. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  2. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  3. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  4. 2023Self-Consuming Generative Models Go MADcited by 239
  5. 2023Self-Alignment with Instruction Backtranslationcited by 149
  6. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  7. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplaycited by 62
  8. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  9. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 54
  10. 2023Fairness-guided Few-shot Prompting for Large Language Modelscited by 53
  11. 2020Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Searchcited by 33
  12. 2023Are Large Language Models Really Robust to Word-Level Perturbations?cited by 32
  13. 2023Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledgecited by 32
  14. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  15. 2023Efficient Toxic Content Detection by Bootstrapping and Distilling Large Language Modelscited by 21

Built on by

Cited by year

Citing papers per year: 2024: 8; 2025: 6.

14 papers.

Sort:

By area:

  1. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  2. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferencescited by 39
  3. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  4. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  5. 2025Safety Reasoning with Guidelinescited by 14
  6. 2025Lifelong Safety Alignment for Language Modelscited by 13
  7. 2024Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Modelscited by 12
  8. 2025Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generationcited by 12
  9. 2024QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learningcited by 10
  10. 2024Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generationcited by 10
  11. 2024MATHWELL: Generating Educational Math Word Problems Using Teacher Annotationscited by 7
  12. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignmentcited by 7
2 more
  1. 2025SHARP: Synthesizing High-quality Aligned Reasoning Problems for Large Reasoning Models Reinforcement Learningcited by 5
  2. 2025Not All Documents Are What You Need for Extracting Instruction Tuning Datacited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samples
  2. 2023Aligning Large Language Models with Human: A Survey
  3. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
  4. 2024Efficient Alignment of Large Language Models via Data Sampling
  5. 2025A Survey on Training-free Alignment of Large Language Models
  6. 2024Bootstrapping Language Models with DPO Implicit Rewards
  7. 2024Aligner: Efficient Alignment by Learning to Correct
  8. 2024Nudging: Inference-time Alignment of LLMs via Guided Decoding