mlsanityData to 7 Oct 2026

Fine-tuning Large Language Models with Sequential Instructions

arXiv 2403.07794 ↗First posted 12 Mar 2024cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Despite the success of existing instruction-tuned models, we find that they usually struggle to respond to queries with multiple instructions. This impairs their performance in complex problems whose solution consists of multiple intermediate tasks. Thus, we contend that part of the fine-tuning data mixture should be sequential--containing a chain of interrelated tasks. We first approach sequential instruction tuning from a task-driven perspective, manually creating interpretable intermediate tasks for multilingual and visual question answering: namely "translate then predict" and "caption then answer". Next, we automate this process by turning instructions in existing datasets (e.g., Alpaca and FlanCoT) into diverse and complex sequential instructions, making our method general-purpose. Models that underwent our sequential instruction tuning show improved results in coding, maths, and open-ended generation. Moreover, we put forward a new benchmark named SeqEval to evaluate a model's ability to follow all the instructions in a sequence, which further corroborates the benefits of our fine-tuning method. We hope that our endeavours will open new research avenues on instruction tuning for complex tasks.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 48 of its 69 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

48 of 69 references.

References · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
36 more
  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2023Mistral 7Bcited by 3,065
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2017Decoupled Weight Decay Regularizationcited by 1,722
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  13. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  14. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  15. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  16. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  17. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  18. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  19. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  20. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  21. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  22. 2018Constituency Parsing with a Self-Attentive Encodercited by 196
  23. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  24. 2018Multilingual Constituency Parsing with Self-Attention and Pre-Trainingcited by 117
  25. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  26. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96
  27. 2023Do Multilingual Language Models Think Better in English?cited by 87
  28. 2023Modular Deep Learningcited by 82
  29. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  30. 2018Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processingcited by 63
  31. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  32. 2023Revisiting Machine Translation for Cross-lingual Classificationcited by 40
  33. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  34. 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 28
  35. 2021Modelling Latent Translations for Cross-Lingual Transfercited by 16
  36. 2024Chain-of-Instructions: Compositional Instruction Tuning on Large Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 8; 2026: 1.

20 papers.

Sort:

By area:

  1. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  2. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  3. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 40
  4. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 34
  5. 2024The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Modelscited by 18
  6. 2024Refiner: Restructure Retrieval Content Efficiently to Advance Question-Answering Capabilitiescited by 16
  7. 2025Can Language Models Follow Multiple Turns of Entangled Instructions?cited by 14
  8. 2025From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuningcited by 12
  9. 2024SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characterscited by 6
  10. 2024Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignmentcited by 6
  11. 2024Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Sciencecited by 4
  12. 2024Compound-QA: A Benchmark for Evaluating LLMs on Compound Questionscited by 4
8 more
  1. 2025QueST: Incentivizing LLMs to Generate Difficult Problemscited by 4
  2. 2025Protecting Vulnerable Voices: Synthetic Dataset Generation for Self-Disclosure Detectioncited by 3
  3. 2024Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebusescited by 2
  4. 2025XL-Suite: Cross-Lingual Synthetic Training and Evaluation Data for Open-Ended Generationcited by 1
  5. 2025When Does Language Transfer Help? Sequential Fine-Tuning for Cross-Lingual Euphemism Detectioncited by 1
  6. 2024FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large Language Modelscited by 0
  7. 2025An Automated LLM-based Pipeline for Asset-Level Database Creation to Assess Deforestation Impactcited by 0
  8. 2026TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuningcited by 0