# IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization

arXiv: https://arxiv.org/abs/2411.06208 · HTML: https://mlsanity.com/paper/2411.06208/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 9 Nov 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [Preference Ranking Optimization for Human Alignment](https://mlsanity.com/paper/2306.17492/) (2023)
- [OpenAGI: When LLM Meets Domain Experts](https://mlsanity.com/paper/2304.04370/) (2023)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://mlsanity.com/paper/2406.17419/) (2024)
- [Wider and Deeper LLM Networks are Fairer LLM Evaluators](https://mlsanity.com/paper/2308.01862/) (2023)
- [When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data](https://mlsanity.com/paper/2102.02201/) (2021)
- [Can Large Language Models Understand Real-World Complex Instructions?](https://mlsanity.com/paper/2309.09150/) (2023)
- [CFBench: A Comprehensive Constraints-Following Benchmark for LLMs](https://mlsanity.com/paper/2408.01122/) (2024)
- [A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration](https://mlsanity.com/paper/2404.00405/) (2024)
- [Learning to Generate Task-Specific Adapters from Task Description](https://mlsanity.com/paper/2101.00420/) (2021)
- [The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trends](https://mlsanity.com/paper/2409.14195/) (2024)
- [Language Models can Evaluate Themselves via Probability Discrepancy](https://mlsanity.com/paper/2405.10516/) (2024)

## Built on by (most cited first, 30 of 31)

- [VerIF: Verification Engineering for Reinforcement Learning in Instruction Following](https://mlsanity.com/paper/2506.09942/) (2025)
- [MM-IFEngine: Towards Multimodal Instruction Following](https://mlsanity.com/paper/2504.07957/) (2025)
- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation](https://mlsanity.com/paper/2510.24821/) (2025)
- [RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following](https://mlsanity.com/paper/2603.25133/) (2026)
- [Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models](https://mlsanity.com/paper/2506.01413/) (2025)
- [SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agents](https://mlsanity.com/paper/2502.15538/) (2025)
- [MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models](https://mlsanity.com/paper/2505.07591/) (2025)
- [SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization](https://mlsanity.com/paper/2505.11166/) (2025)
- [EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving](https://mlsanity.com/paper/2506.02672/) (2025)
- [IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation](https://mlsanity.com/paper/2603.04738/) (2026)
- [Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs](https://mlsanity.com/paper/2505.14530/) (2025)
- [EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models](https://mlsanity.com/paper/2506.08375/) (2025)
- [Routine: A Structural Planning Framework for LLM Agent System in Enterprise](https://mlsanity.com/paper/2507.14447/) (2025)
- [Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following](https://mlsanity.com/paper/2512.23457/) (2025)
- [IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2508.04632/) (2025)
- [Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following](https://mlsanity.com/paper/2508.03178/) (2025)
- [IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation](https://mlsanity.com/paper/2511.01014/) (2025)
- [Reverse Preference Optimization for Complex Instruction Following](https://mlsanity.com/paper/2505.22172/) (2025)
- [InstructEngine: Instruction-driven Text-to-Image Alignment](https://mlsanity.com/paper/2504.10329/) (2025)
- [DecIF: Improving Instruction-Following through Meta-Decomposition](https://mlsanity.com/paper/2505.13990/) (2025)
- [CCR-Bench: A Comprehensive Benchmark for Evaluating LLMs on Complex Constraints, Control Flows, and Real-World Cases](https://mlsanity.com/paper/2603.07886/) (2026)
- [Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design](https://mlsanity.com/paper/2603.12826/) (2026)
- [One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness](https://mlsanity.com/paper/2604.13006/) (2026)
- [SEIF: Self-Evolving Reinforcement Learning for Instruction Following](https://mlsanity.com/paper/2605.07465/) (2026)
- [Knots: A Large-Scale Multi-Agent Enhanced Expert-Annotated Dataset and LLM Prompt Optimization for NOTAM Semantic Parsing](https://mlsanity.com/paper/2511.12630/) (2025)
- [STAIF: A Stage-wise Optimization for Complex Instruction Following](https://mlsanity.com/paper/2607.22649/) (2026)
- [UNSPECIFIC: General Constraint Synthesis for Breaking Copy-and-Paste Shortcut in LLM Instruction Following](https://mlsanity.com/paper/2608.09154/) (2026)
- [Cross-Relational Preference Learning for Better LLM Instruction Following](https://mlsanity.com/paper/2608.29352/) (2026)
- [ScopeIF: Improving Scope-Aware Precise Instruction-Following in Large Language Models via Graded Reward Modeling](https://mlsanity.com/paper/2609.32189/) (2026)

## Abstract (arXiv)

In the realm of large language models (LLMs), the ability of models to accurately follow instructions is paramount as more agents and applications leverage LLMs for construction, where the complexity of instructions are rapidly increasing. However, on the one hand, there is only a certain amount of complex instruction evaluation data; on the other hand, there are no dedicated algorithms to improve the ability to follow complex instructions. To this end, this paper introduces TRACE, a benchmark for improving and evaluating the complex instructionfollowing ability, which consists of 120K training data and 1K evaluation data. Furthermore, we propose IOPO (Input-Output Preference Optimization) alignment method which takes both input and output preference pairs into consideration, where LLMs not only rapidly align with response preferences but also meticulously explore the instruction preferences. Extensive experiments on both in-domain and outof-domain datasets confirm the effectiveness of IOPO, showing 8.15%, 2.18% improvements on in-domain data and 6.29%, 3.13% on outof-domain data compared to SFT and DPO respectively.