# CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions

arXiv: https://arxiv.org/abs/2410.03077 · HTML: https://mlsanity.com/paper/2410.03077/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 29 of 29)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://mlsanity.com/paper/2212.12017/) (2022)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [Fine-tuned Language Models are Continual Learners](https://mlsanity.com/paper/2205.12393/) (2022)
- [Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning](https://mlsanity.com/paper/2402.04833/) (2024)
- [Contrastive Vision-Language Pre-training with Limited Resources](https://mlsanity.com/paper/2112.09331/) (2021)
- [A Two-Stage Adaptation of Large Language Models for Text Ranking](https://mlsanity.com/paper/2311.16720/) (2023)
- [Specialist or Generalist? Instruction Tuning for Specific NLP Tasks](https://mlsanity.com/paper/2310.15326/) (2023)
- [CITING: Large Language Models Create Curriculum for Instruction Tuning](https://mlsanity.com/paper/2310.02527/) (2023)
- [TaskWeb: Selecting Better Source Tasks for Multi-task NLP](https://mlsanity.com/paper/2305.13256/) (2023)

## Built on by (most cited first, 14 of 14)

- [APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Training](https://mlsanity.com/paper/2506.03483/) (2025)
- [Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models](https://mlsanity.com/paper/2507.18504/) (2025)
- [Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs](https://mlsanity.com/paper/2503.07663/) (2025)
- [Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation](https://mlsanity.com/paper/2510.15304/) (2025)
- [DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization](https://mlsanity.com/paper/2411.14055/) (2024)
- [AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs](https://mlsanity.com/paper/2507.18584/) (2025)
- [SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe](https://mlsanity.com/paper/2410.05248/) (2024)
- [From Past To Path: Masked History Learning for Next-Item Prediction in Generative Recommendation](https://mlsanity.com/paper/2509.23649/) (2025)
- [GradientSpace: Unsupervised Data Clustering for Improved Instruction Tuning](https://mlsanity.com/paper/2512.06678/) (2025)
- [Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling](https://mlsanity.com/paper/2604.18264/) (2026)
- [MDIT-Bench: Evaluating the Dual-Implicit Toxicity in Large Multimodal Models](https://mlsanity.com/paper/2505.17144/) (2025)
- [CDT: A Comprehensive Capability Framework for Large Language Models Across Cognition, Domain, and Task](https://mlsanity.com/paper/2509.24422/) (2025)
- [SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models](https://mlsanity.com/paper/2509.24781/) (2025)
- [Conflicting Supervision Moves Commitment, Not Capability: A 12.29σ arrangement effect that is exactly zero under a convention-agnostic score](https://mlsanity.com/paper/2610.00234/) (2026)

## Abstract (arXiv)

With instruction tuning, Large Language Models (LLMs) can enhance their ability to adhere to commands. Diverging from most works focusing on data mixing, our study concentrates on enhancing the model's capabilities from the perspective of data sampling during training. Drawing inspiration from the human learning process, where it is generally easier to master solutions to similar topics through focused practice on a single type of topic, we introduce a novel instruction tuning strategy termed CommonIT: Commonality-aware Instruction Tuning. Specifically, we cluster instruction datasets into distinct groups with three proposed metrics (Task, Embedding and Length). We ensure each training mini-batch, or "partition", consists solely of data from a single group, which brings about both data randomness across mini-batches and intra-batch data similarity. Rigorous testing on LLaMa models demonstrates CommonIT's effectiveness in enhancing the instruction-following capabilities of LLMs through IT datasets (FLAN, CoT, and Alpaca) and models (LLaMa2-7B, Qwen2-7B, LLaMa 13B, and BLOOM 7B). CommonIT consistently boosts an average improvement of 2.1% on the general domain (i.e., the average score of Knowledge, Reasoning, Multilinguality and Coding) with the Length metric, and 5.2% on the special domain (i.e., GSM, Openfunctions and Code) with the Task metric, and 3.8% on the specific tasks (i.e., MMLU) with the Embedding metric. Code is available at urlhttps://github.com/raojay7/CommonIT.