# TAG-INSTRUCT: Controlled Instruction Complexity Enhancement through Structure-based Augmentation

arXiv: https://arxiv.org/abs/2505.18557 · HTML: https://mlsanity.com/paper/2505.18557/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 24 May 2025.

## Builds on (most cited first, 29 of 29)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [Distilling System 2 into System 1](https://mlsanity.com/paper/2407.06023/) (2024)
- [QuRating: Selecting High-Quality Data for Training Language Models](https://mlsanity.com/paper/2402.09739/) (2024)
- [Nemotron-4 340B Technical Report](https://mlsanity.com/paper/2406.11704/) (2024)
- [MoDS: Model-oriented Data Selection for Instruction Tuning](https://mlsanity.com/paper/2311.15653/) (2023)
- [Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning](https://mlsanity.com/paper/2402.13669/) (2024)
- [Large Concept Models: Language Modeling in a Sentence Representation Space](https://mlsanity.com/paper/2412.08821/) (2024)
- [Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning](https://mlsanity.com/paper/2402.04833/) (2024)
- [Planning In Natural Language Improves LLM Search For Code Generation](https://mlsanity.com/paper/2409.03733/) (2024)
- [TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation](https://mlsanity.com/paper/2407.10114/) (2024)
- [CodecLM: Aligning Language Models with Tailored Synthetic Data](https://mlsanity.com/paper/2404.05875/) (2024)
- [Rethinking Data Selection for Supervised Fine-Tuning](https://mlsanity.com/paper/2402.06094/) (2024)
- [FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs Only](https://mlsanity.com/paper/2408.01323/) (2024)
- [SentenceVAE: Enable Next-sentence Prediction for Large Language Models with Faster Speed, Higher Accuracy and Longer Context](https://mlsanity.com/paper/2408.00655/) (2024)

## Built on by (most cited first, 3 of 3)

- [LM-mixup: Text Data Augmentation via Language Model based Mixup](https://mlsanity.com/paper/2510.20449/) (2025)
- [InstructDiff: Domain-Adaptive Data Selection via Differential Entropy for Efficient LLM Fine-Tuning](https://mlsanity.com/paper/2601.23006/) (2026)
- [Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL](https://mlsanity.com/paper/2607.03833/) (2026)

## Abstract (arXiv)

High-quality instruction data is crucial for developing large language models (LLMs), yet existing approaches struggle to effectively control instruction complexity. We present TAG-INSTRUCT, a novel framework that enhances instruction complexity through structured semantic compression and controlled difficulty augmentation. Unlike previous prompt-based methods operating on raw text, TAG-INSTRUCT compresses instructions into a compact tag space and systematically enhances complexity through RL-guided tag expansion. Through extensive experiments, we show that TAG-INSTRUCT outperforms existing instruction complexity augmentation approaches. Our analysis reveals that operating in tag space provides superior controllability and stability across different instruction synthesis frameworks.