# InfiAlign: A Scalable and Sample-Efficient Framework for Aligning LLMs to Enhance Reasoning Capabilities

arXiv: https://arxiv.org/abs/2508.05496 · HTML: https://mlsanity.com/paper/2508.05496/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 21 of 21)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond](https://mlsanity.com/paper/2503.10460/) (2025)
- [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025)
- [Skywork Open Reasoner 1 Technical Report](https://mlsanity.com/paper/2505.22312/) (2025)
- [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022)
- [Llama-Nemotron: Efficient Reasoning Models](https://mlsanity.com/paper/2505.00949/) (2025)
- [OpenCodeReasoning: Advancing Data Distillation for Competitive Coding](https://mlsanity.com/paper/2504.01943/) (2025)
- [AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning](https://mlsanity.com/paper/2505.16400/) (2025)
- [Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation](https://mlsanity.com/paper/2402.18191/) (2024)
- [Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning](https://mlsanity.com/paper/2311.08182/) (2023)
- [1.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Training](https://mlsanity.com/paper/2503.19633/) (2025)
- [Entropy Law: The Story Behind Data Compression and LLM Performance](https://mlsanity.com/paper/2407.06645/) (2024)
- [AlpaGasus: Training A Better Alpaca with Fewer Data](https://mlsanity.com/paper/2307.08701/) (2023)
- [Selection of LLM Fine-Tuning Data based on Orthogonal Rules](https://mlsanity.com/paper/2410.04715/) (2024)
- [360-LLaMA-Factory: Plug & Play Sequence Parallelism for Long Post-Training](https://mlsanity.com/paper/2505.22296/) (2025)

## Built on by (most cited first, 1 of 1)

- [A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models](https://mlsanity.com/paper/2509.22536/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have exhibited impressive reasoning abilities on a wide range of complex tasks. However, enhancing these capabilities through post-training remains resource intensive, particularly in terms of data and computational cost. Although recent efforts have sought to improve sample efficiency through selective data curation, existing methods often rely on heuristic or task-specific strategies that hinder scalability. In this work, we introduce InfiAlign, a scalable and sample-efficient post-training framework that integrates supervised fine-tuning (SFT) with Direct Preference Optimization (DPO) to align LLMs for enhanced reasoning. At the core of InfiAlign is a robust data selection pipeline that automatically curates high-quality alignment data from open-source reasoning datasets using multidimensional quality metrics. This pipeline enables significant performance gains while drastically reducing data requirements and remains extensible to new data sources. When applied to the Qwen2.5-Math-7B-Base model, our SFT model achieves performance on par with DeepSeek-R1-Distill-Qwen-7B, while using only approximately 12% of the training data, and demonstrates strong generalization across diverse reasoning tasks. Additional improvements are obtained through the application of DPO, with particularly notable gains in mathematical reasoning tasks. The model achieves an average improvement of 3.89% on AIME 24/25 benchmarks. Our results highlight the effectiveness of combining principled data selection with full-stage post-training, offering a practical solution for aligning large reasoning models in a scalable and data-efficient manner. The model checkpoints are available at https://huggingface.co/InfiX-ai/InfiAlign-Qwen-7B-SFT.