# Instruction Tuning With Loss Over Instructions

arXiv: https://arxiv.org/abs/2405.14394 · HTML: https://mlsanity.com/paper/2405.14394/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 80 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 30 of 58)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)

## Built on by (most cited first, 30 of 40)

- [A Closer Look at Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2410.08109/) (2024)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [Navigating Rifts in Human-LLM Grounding: Study and Benchmark](https://mlsanity.com/paper/2503.13975/) (2025)
- [Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface](https://mlsanity.com/paper/2501.09798/) (2025)
- [Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation](https://mlsanity.com/paper/2410.18565/) (2024)
- [Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQL](https://mlsanity.com/paper/2504.15077/) (2025)
- [Instruction Following without Instruction Tuning](https://mlsanity.com/paper/2409.14254/) (2024)
- [Understanding the Role of User Profile in the Personalization of Large Language Models](https://mlsanity.com/paper/2406.17803/) (2024)
- [MDCure: A Scalable Pipeline for Multi-Document Instruction-Following](https://mlsanity.com/paper/2410.23463/) (2024)
- [Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?](https://mlsanity.com/paper/2509.04292/) (2025)
- [Toward Secure Tuning: Mitigating Security Risks from Instruction Fine-Tuning](https://mlsanity.com/paper/2410.04524/) (2024)
- [Learning to Instruct for Visual Instruction Tuning](https://mlsanity.com/paper/2503.22215/) (2025)
- [Understanding Likelihood Over-optimisation in Direct Alignment Algorithms](https://mlsanity.com/paper/2410.11677/) (2024)
- [Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence](https://mlsanity.com/paper/2506.23908/) (2025)
- [Bielik 11B v3: Multilingual Large Language Model for European Languages](https://mlsanity.com/paper/2601.11579/) (2025)
- [Bielik v3 Small: Technical Report](https://mlsanity.com/paper/2505.02550/) (2025)
- [SEPS: A Separability Measure for Robust Unlearning in LLMs](https://mlsanity.com/paper/2505.14832/) (2025)
- [Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation](https://mlsanity.com/paper/2512.21002/) (2025)
- [Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide](https://mlsanity.com/paper/2411.09539/) (2024)
- [All-in-One Tuning and Structural Pruning for Domain-Specific LLMs](https://mlsanity.com/paper/2412.14426/) (2024)
- [Toward Cybersecurity-Expert Small Language Models](https://mlsanity.com/paper/2510.14113/) (2025)
- [DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures](https://mlsanity.com/paper/2605.10770/) (2026)
- [SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe](https://mlsanity.com/paper/2410.05248/) (2024)
- [Got Compute, but No Data: Lessons From Post-training a Finnish LLM](https://mlsanity.com/paper/2503.09407/) (2025)
- [Learning from the Undesirable: Robust Adaptation of Language Models without Forgetting](https://mlsanity.com/paper/2511.13052/) (2025)
- [Autonomous Knowledge Graph Exploration with Adaptive Breadth-Depth Retrieval](https://mlsanity.com/paper/2601.13969/) (2026)
- [Bielik 11B v2 Technical Report](https://mlsanity.com/paper/2505.02410/) (2025)
- [Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky](https://mlsanity.com/paper/2507.03336/) (2025)
- [Review of Hallucination Understanding in Large Language and Vision Models](https://mlsanity.com/paper/2510.00034/) (2025)
- [Improving Multilingual Math Reasoning for African Languages](https://mlsanity.com/paper/2505.19848/) (2025)

## Abstract (arXiv)

Instruction tuning plays a crucial role in shaping the outputs of language models (LMs) to desired styles. In this work, we propose a simple yet effective method, Instruction Modelling (IM), which trains LMs by applying a loss function to the instruction and prompt part rather than solely to the output part. Through experiments across 21 diverse benchmarks, we show that, in many scenarios, IM can effectively improve the LM performance on both NLP tasks (e.g., MMLU, TruthfulQA, and HumanEval) and open-ended generation benchmarks (e.g., MT-Bench and AlpacaEval). Remarkably, in the most advantageous case, IM boosts model performance on AlpacaEval 1.0 by over 100%. We identify two key factors influencing the effectiveness of IM: (1) The ratio between instruction length and output length in the training data; and (2) The number of training examples. We observe that IM is especially beneficial when trained on datasets with lengthy instructions paired with brief outputs, or under the Superficial Alignment Hypothesis (SAH) where a small amount of training examples are used for instruction tuning. Further analysis substantiates our hypothesis that our improvement can be attributed to reduced overfitting to instruction tuning datasets. It is worth noting that we are not proposing ours as a replacement for current fine-tuning processes. Instead, our work aims to provide practical guidance for instruction tuning LMs, especially in low-resource scenarios.