# Following Length Constraints in Instructions

arXiv: https://arxiv.org/abs/2406.17744 · HTML: https://mlsanity.com/paper/2406.17744/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [Disentangling Length from Quality in Direct Preference Optimization](https://mlsanity.com/paper/2403.19159/) (2024)
- [ODIN: Disentangled Reward Mitigates Hacking in RLHF](https://mlsanity.com/paper/2402.07319/) (2024)
- [Controllable Abstractive Summarization](https://mlsanity.com/paper/1711.05217/) (2017)
- [Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning](https://mlsanity.com/paper/2402.04833/) (2024)
- [Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.05199/) (2023)
- [Prompt-Based Length Controlled Generation with Reinforcement Learning](https://mlsanity.com/paper/2308.12030/) (2023)

## Built on by (most cited first, 30 of 46)

- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025)
- [Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge](https://mlsanity.com/paper/2407.19594/) (2024)
- [LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs](https://mlsanity.com/paper/2408.07055/) (2024)
- [Thinking LLMs: General Instruction Following with Thought Generation](https://mlsanity.com/paper/2410.10630/) (2024)
- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.24377/) (2025)
- [Scalable Chain of Thoughts via Elastic Reasoning](https://mlsanity.com/paper/2505.05315/) (2025)
- [A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications](https://mlsanity.com/paper/2410.15595/) (2024)
- [From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium](https://mlsanity.com/paper/2506.08292/) (2025)
- [Train Long, Think Short: Curriculum Learning for Efficient Reasoning](https://mlsanity.com/paper/2508.08940/) (2025)
- [An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint](https://mlsanity.com/paper/2504.14350/) (2025)
- [Reasoning Beyond Limits: Advances and Open Problems for LLMs](https://mlsanity.com/paper/2503.22732/) (2025)
- [Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement](https://mlsanity.com/paper/2505.07961/) (2025)
- [R.I.P.: Better Models by Survival of the Fittest Prompts](https://mlsanity.com/paper/2501.18578/) (2025)
- [On Reasoning Strength Planning in Large Reasoning Models](https://mlsanity.com/paper/2506.08390/) (2025)
- [The Era of Real-World Human Interaction: RL from User Conversations](https://mlsanity.com/paper/2509.25137/) (2025)
- [Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following](https://mlsanity.com/paper/2502.17204/) (2025)
- [Zero-Shot Strategies for Length-Controllable Summarization](https://mlsanity.com/paper/2501.00233/) (2024)
- [Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts](https://mlsanity.com/paper/2412.04628/) (2024)
- [Length Desensitization in Direct Preference Optimization](https://mlsanity.com/paper/2409.06411/) (2024)
- [Length Controlled Generation for Black-box LLMs](https://mlsanity.com/paper/2412.14656/) (2024)
- [Med-U1: Incentivizing Unified Medical Reasoning in LLMs via Large-scale Reinforcement Learning](https://mlsanity.com/paper/2506.12307/) (2025)
- [mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval](https://mlsanity.com/paper/2501.19264/) (2025)
- [Revisiting the Reliability of Language Models in Instruction-Following](https://mlsanity.com/paper/2512.14754/) (2025)
- [Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling](https://mlsanity.com/paper/2502.00814/) (2025)
- [Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions](https://mlsanity.com/paper/2601.05414/) (2026)
- [AMPO: Active Multi-Preference Optimization for Self-play Preference Selection](https://mlsanity.com/paper/2502.18293/) (2025)
- [MedReadCtrl: Personalizing medical text generation with readability-controlled instruction learning](https://mlsanity.com/paper/2507.07419/) (2025)
- [Can LLMs Track Their Output Length? A Dynamic Feedback Mechanism for Precise Length Regulation](https://mlsanity.com/paper/2601.01768/) (2026)
- [Neuro-Symbolic Verification on Instruction Following of LLMs](https://mlsanity.com/paper/2601.17789/) (2026)

## Abstract (arXiv)

Aligned instruction following models can better fulfill user requests than their unaligned counterparts. However, it has been shown that there is a length bias in evaluation of such models, and that training algorithms tend to exploit this bias by learning longer responses. In this work we show how to train models that can be controlled at inference time with instructions containing desired length constraints. Such models are superior in length instructed evaluations, outperforming standard instruction following models such as GPT4, Llama 3 and Mixtral.