mlsanityData to 14 Oct 2026

Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models

arXiv 2506.11116 ↗First posted 9 Jun 2025cs.CL cs.AIcited by 50

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) demonstrate strong performance in real-world applications, yet existing open-source instruction datasets often concentrate on narrow domains, such as mathematics or coding, limiting generalization and widening the gap with proprietary models. To bridge this gap, we introduce Infinity-Instruct, a high-quality instruction dataset designed to enhance both foundational and chat capabilities of LLMs through a two-phase pipeline. In Phase 1, we curate 7.4M high-quality foundational instructions (InfInstruct-F-7.4M) from over 100M samples using hybrid data selection techniques. In Phase 2, we synthesize 1.5M high-quality chat instructions (InfInstruct-G-1.5M) through a two-stage process involving instruction selection, evolution, and diagnostic filtering. We empirically evaluate Infinity-Instruct by fine-tuning several open-source models, including Mistral, LLaMA, Qwen, and Yi, and observe substantial performance gains across both foundational and instruction following benchmarks, consistently surpassing official instruction-tuned counterparts. Notably, InfInstruct-LLaMA3.1-70B outperforms GPT-4-0314 by 8.6% on instruction following tasks while achieving comparable foundational performance. These results underscore the synergy between foundational and chat training and offer new insights into holistic LLM development. Our datasetfootnotehttps://huggingface.co/datasets/BAAI/Infinity-Instruct and codesfootnotehttps://gitee.com/li-touch/infinity-instruct have been publicly released.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

23 of 31 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
11 more
  1. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  2. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  3. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  4. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  5. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  6. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
  7. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  8. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  9. 2024InfinityMATH: A Scalable Instruction Tuning Dataset in Programmatic Mathematical Reasoningcited by 15
  10. 2024Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependencycited by 13
  11. 2024Aquila2 Technical Reportcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 31.

50 papers.

Sort:

By area:

  1. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  2. 2025DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AIcited by 23
  3. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  4. 2025Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Modelscited by 20
  5. 2026ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selectioncited by 18
  6. 2026LK Losses: Direct Acceptance Rate Optimization for Speculative Decodingcited by 18
  7. 2025ShareChat: A Dataset of Chatbot Conversations in the Wildcited by 15
  8. 2025FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Trainingcited by 11
  9. 2025Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuningcited by 8
  10. 2025RedOne: Revealing Domain-specific LLM Post-Training in Social Networking Servicescited by 6
  11. 2025Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Followingcited by 6
  12. 2026Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Modelcited by 6
38 more
  1. 2025SpikingBrain: Spiking Brain-inspired Large Modelscited by 5
  2. 2025PersonaFuse: A Personality Activation-Driven Framework for Enhancing Human-LLM Interactionscited by 5
  3. 2025Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Datacited by 5
  4. 2026Towards Compositional Generalization of LLMs via Skill Taxonomy Guided Data Synthesiscited by 4
  5. 2025AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretrainingcited by 3
  6. 2025Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Followingcited by 3
  7. 2025Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillationcited by 3
  8. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  9. 2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
  10. 2026From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizonscited by 3
  11. 2025LexInstructEval: Lexical Instruction Following Evaluation for Large Language Modelscited by 2
  12. 2026EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillationcited by 2
  13. 2026K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMscited by 2
  14. 2026DataFoundry: Evolving Data Preparators via Recursive Self-Improvementcited by 2
  15. 2025Linear Preference Optimization: Decoupled Gradient Control via Absolute Regularizationcited by 1
  16. 2025GIFT: Group-Relative Implicit Fine-Tuning Integrates GRPO with DPO and UNAcited by 1
  17. 2026Kakugo: Distillation of Low-Resource Languages into Small Language Modelscited by 1
  18. 2026CCR-Bench: A Comprehensive Benchmark for Evaluating LLMs on Complex Constraints, Control Flows, and Real-World Casescited by 1
  19. 2026Discovering Decoupled Functional Modules in Large Language Modelscited by 1
  20. 2026VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singingcited by 1
  21. 2026DataPrep-Bench: Benchmarking LLMs as Training Data Preparatorscited by 1
  22. 2025Input-Time Scaling: Adding Noise and Irrelevance into Less-Is-More Drastically Improves Reasoning Performance and Efficiencycited by 0
  23. 2026From Tags to Trees: Structuring Fine-Grained Knowledge for Controllable Data Selection in LLM Instruction Tuningcited by 0
  24. 2026MAR: Efficient Large Language Models via Module-aware Architecture Refinementcited by 0
  25. 2026ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patternscited by 0
  26. 2026Language Model Maps for Prompt-Response Distributions via Log-Likelihood Vectorscited by 0
  27. 2026PROMPT2BOX:Improving LLM Weakness Discovery and Specificity Estimation by Uncovering Entailment Structure among Promptscited by 0
  28. 2026SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decodingcited by 0
  29. 2026Language Generation as Optimal Control: Closed-Loop Diffusion in Latent Control Spacecited by 0
  30. 2026Multi-Token Residual Predictioncited by 0
  31. 2026The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Workscited by 0
  32. 2026mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Modelscited by 0
  33. 2026Enhancing Fitness Intelligence through Domain-Specific LLM Post-Trainingcited by 0
  34. 2026CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioningcited by 0
  35. 2026Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Modelscited by 0
  36. 2026Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMscited by 0
  37. 2026Adaptive Mutual Distillation for Balanced Multi-Task Post-Training of Large Language Modelscited by 0
  38. 2026Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Expertscited by 0