mlsanityData to 14 Oct 2026

A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment

arXiv 2308.05696 ↗First posted 10 Aug 2023cs.CLcited by 33

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Training large language models (LLMs) with open-domain instruction data has yielded remarkable success in aligning to end tasks and human preferences. Extensive research has highlighted the importance of the quality and diversity of instruction data. However, the impact of data complexity, as a crucial metric, remains relatively unexplored from three aspects: (1)where the sustainability of performance improvements with increasing complexity is uncertain; (2)whether the improvement brought by complexity merely comes from introducing more training tokens; and (3)where the potential benefits of incorporating instructions from easy to difficult are not yet fully understood. In this paper, we propose Tree-Instruct to systematically enhance the instruction complexity in a controllable manner. By adding a specified number of nodes to instructions' semantic trees, this approach not only yields new instruction data from the modified tree but also allows us to control the difficulty level of modified instructions. Our preliminary experiments reveal the following insights: (1)Increasing complexity consistently leads to sustained performance improvements of LLMs. (2)Under the same token budget, a few complex instructions outperform diverse yet simple instructions. (3)Curriculum instruction tuning might not yield the anticipated results; focusing on increasing complexity appears to be the key.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 64 of its 105 references that are in the map. First posted on arXiv 10 Aug 2023.

Builds on

64 of 105 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Preference Ranking Optimization for Human Alignment30 Jun
  2. 2023LIMA: Less Is More for Alignment18 May
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
38 more
  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  3. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  7. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  13. 2023Instruction Tuning with GPT-4cited by 686
  14. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  15. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  16. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  17. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  18. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  19. 2023Scaling Data-Constrained Language Modelscited by 403
  20. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  21. 2020Multi-modal Transformer for Video Retrievalcited by 395
  22. 2022Self-critiquing models for assisting human evaluatorscited by 380
  23. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  24. 2023GPTScore: Evaluate as You Desirecited by 361
  25. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  26. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  27. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 255
  28. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  29. 2014Object Detectors Emerge in Deep Scene CNNscited by 179
  30. 2020Support-set bottlenecks for video-text representation learningcited by 170
  31. 2015Visualizing and Understanding Recurrent Networkscited by 160
  32. 2017Learning to Generate Reviews and Discovering Sentimentcited by 148
  33. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
  34. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 94
  35. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  36. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  37. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  38. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 15; 2025: 12; 2026: 3.

33 papers.

Sort:

By area:

  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  3. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  5. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  6. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 77
  7. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  8. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selectioncited by 48
  9. 2023One-Shot Learning as Instruction Data Prospector for Large Language Modelscited by 43
  10. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  11. 2024Improving Data Efficiency via Curating LLM-Driven Rating Systemscited by 29
  12. 2024What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?cited by 23
21 more
  1. 2024SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuningcited by 19
  2. 2024Intelligence at the Edge of Chaoscited by 13
  3. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  4. 2025From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuningcited by 12
  5. 2025MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Spacecited by 12
  6. 2025Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learningcited by 7
  7. 2024Optimizing Instruction Synthesis: Effective Exploration of Evolutionary Space with Tree Searchcited by 5
  8. 2024TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learningcited by 4
  9. 2025MDIT: A Model-free Data Interpolation Method for Diverse Instruction Tuningcited by 3
  10. 2025Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lenscited by 3
  11. 2024CoEvol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperationcited by 2
  12. 2024AIDE: Attribute-Guided MultI-Hop Data Expansion for Data Scarcity in Task-Specific Fine-tuningcited by 2
  13. 2025HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Modelscited by 2
  14. 2025ClusterUCB: Efficient Gradient-Based Data Selection for Targeted Fine-Tuning of LLMscited by 2
  15. 2025Teaching According to Talents! Instruction Tuning LLMs with Competence-Aware Curriculum Learningcited by 2
  16. 2025Call for Rigor in Reporting Quality of Instruction Tuning Datacited by 1
  17. 2025LASER: Stratified Selective Sampling for Instruction Tuning with Dedicated Scoring Strategycited by 1
  18. 2026From Instance Selection to Fixed-Pool Data Recipe Search for Supervised Fine-Tuningcited by 1
  19. 2025TACOS: Open Tagging and Comparative Scoring for Instruction Fine-Tuning Data Selectioncited by 0
  20. 2026Synthesizing Instruction-Tuning Datasets with Contrastive Decodingcited by 0
  21. 2026Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  2. 2026Operationalising the Superficial Alignment Hypothesis via Task Complexity
  3. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  4. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
  5. 2023Dynamics of Instruction Fine-Tuning for Chinese Large Language Models
  6. 2025Aligning Instruction Tuning with Pre-training
  7. 2025MAIN: Mutual Alignment Is Necessary for instruction tuning
  8. 2024Diverse and Fine-Grained Instruction-Following Ability Exploration with Synthetic Data