# DIVE: Diversified Iterative Self-Improvement

arXiv: https://arxiv.org/abs/2501.00747 · HTML: https://mlsanity.com/paper/2501.00747/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 1 Jan 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-rewarding alignment and self-training](https://mlsanity.com/synth/self-rewarding-alignment-and-self-training/)

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [SLiC-HF: Sequence Likelihood Calibration with Human Feedback](https://mlsanity.com/paper/2305.10425/) (2023)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [KTO: Model Alignment as Prospect Theoretic Optimization](https://mlsanity.com/paper/2402.01306/) (2024)
- [Iterative Reasoning Preference Optimization](https://mlsanity.com/paper/2404.19733/) (2024)
- [Making Large Language Models Better Reasoners with Step-Aware Verifier](https://mlsanity.com/paper/2206.02336/) (2022)
- [Orca-Math: Unlocking the potential of SLMs in Grade School Math](https://mlsanity.com/paper/2402.14830/) (2024)
- [Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data](https://mlsanity.com/paper/2404.01413/) (2024)
- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [Evaluating the Evaluation of Diversity in Natural Language Generation](https://mlsanity.com/paper/2004.02990/) (2020)
- [Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss](https://mlsanity.com/paper/2312.16682/) (2023)
- [NV-Retriever: Improving text embedding models with effective hard-negative mining](https://mlsanity.com/paper/2407.15831/) (2024)
- [A Simple, Fast Diverse Decoding Algorithm for Neural Generation](https://mlsanity.com/paper/1611.08562/) (2016)
- [ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline](https://mlsanity.com/paper/2404.02893/) (2024)
- [Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples](https://mlsanity.com/paper/2406.05673/) (2024)
- [Progress or Regress? Self-Improvement Reversal in Post-training](https://mlsanity.com/paper/2407.05013/) (2024)

## Built on by (most cited first, 3 of 3)

- [Diverse Preference Optimization](https://mlsanity.com/paper/2501.18101/) (2025)
- [Self-Improvements in Modern Agentic Systems: A Survey](https://mlsanity.com/paper/2607.13104/) (2026)
- [Unlocking LLM Creativity in Science through Analogical Reasoning](https://mlsanity.com/paper/2605.11258/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have demonstrated the effectiveness of Iterative Self-Improvement (ISI) techniques. However, continuous training on self-generated data leads to reduced output diversity, a limitation particularly critical in reasoning tasks where diverse solution paths are essential. We present DIVE (Diversified Iterative Self-Improvement), a novel framework that addresses this challenge through two key components: Sample Pool Expansion for broader solution exploration, and Data Selection for balancing diversity and quality in preference pairs. Experiments on MATH and GSM8k datasets show that DIVE achieves a 10% to 45% relative increase in output diversity metrics while maintaining performance quality compared to vanilla ISI. Our ablation studies confirm both components' significance in achieving these improvements. Code is available at https://github.com/qinyiwei/DIVE.