# How Instruction-Tuning Imparts Length Control: A Cross-Lingual Mechanistic Analysis

arXiv: https://arxiv.org/abs/2509.02075 · HTML: https://mlsanity.com/paper/2509.02075/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 2 Sep 2025.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Interpreting Language Models with Contrastive Explanations](https://mlsanity.com/paper/2202.10419/) (2022)
- [Evaluating Large Language Models on Controlled Generation Tasks](https://mlsanity.com/paper/2310.14542/) (2023)
- [COLLIE: Systematic Construction of Constrained Text Generation Tasks](https://mlsanity.com/paper/2307.08689/) (2023)
- [Controllable Text Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2408.12599/) (2024)
- [From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning](https://mlsanity.com/paper/2310.00492/) (2023)
- [Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning](https://mlsanity.com/paper/2305.11383/) (2023)
- [Benchmarking Large Language Models on Controllable Generation under Diversified Instructions](https://mlsanity.com/paper/2401.00690/) (2024)
- [On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Task](https://mlsanity.com/paper/2410.06496/) (2024)
- [Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention](https://mlsanity.com/paper/2310.19084/) (2023)

## Built on by (most cited first, 1 of 1)

- [ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning](https://mlsanity.com/paper/2510.18250/) (2025)

## Abstract (arXiv)

Adhering to explicit length constraints, such as generating text with a precise word count, remains a significant challenge for Large Language Models (LLMs). This study aims at investigating the differences between foundation models and their instruction-tuned counterparts, on length-controlled text generation in English and Italian. We analyze both performance and internal component contributions using Cumulative Weighted Attribution, a metric derived from Direct Logit Attribution. Our findings reveal that instruction-tuning substantially improves length control, primarily by specializing components in deeper model layers. Specifically, attention heads in later layers of IT models show increasingly positive contributions, particularly in English. In Italian, while attention contributions are more attenuated, final-layer MLPs exhibit a stronger positive role, suggesting a compensatory mechanism. These results indicate that instruction-tuning reconfigures later layers for task adherence, with component-level strategies potentially adapting to linguistic context.