# Cross-Relational Preference Learning for Better LLM Instruction Following

arXiv: https://arxiv.org/abs/2608.29352 · HTML: https://mlsanity.com/paper/2608.29352/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 29 Aug 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2308.01825/) (2023)
- [Generalizing Verifiable Instruction Following](https://mlsanity.com/paper/2507.02833/) (2025)
- [FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models](https://mlsanity.com/paper/2310.20410/) (2023)
- [Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models](https://mlsanity.com/paper/2406.13542/) (2024)
- [VerIF: Verification Engineering for Reinforcement Learning in Instruction Following](https://mlsanity.com/paper/2506.09942/) (2025)
- [Suri: Multi-constraint Instruction Following for Long-form Text Generation](https://mlsanity.com/paper/2406.19371/) (2024)
- [Large Language Model Instruction Following: A Survey of Progresses and Challenges](https://mlsanity.com/paper/2303.10475/) (2023)
- [IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization](https://mlsanity.com/paper/2411.06208/) (2024)
- [Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Models](https://mlsanity.com/paper/2404.02823/) (2024)
- [From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models](https://mlsanity.com/paper/2404.15846/) (2024)
- [SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models](https://mlsanity.com/paper/2412.11605/) (2024)
- [UltraIF: Advancing Instruction Following from the Wild](https://mlsanity.com/paper/2502.04153/) (2025)
- [Contrastive Instruction Tuning](https://mlsanity.com/paper/2402.11138/) (2024)
- [Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models](https://mlsanity.com/paper/2501.04945/) (2025)
- [MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training](https://mlsanity.com/paper/2502.11541/) (2025)
- [Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization](https://mlsanity.com/paper/2408.07471/) (2024)
- [Metric Calculating Benchmark: Code-Verifiable Complicate Instruction Following Benchmark for Large Language Models](https://mlsanity.com/paper/2510.07892/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) still exhibit limited capability in following complex instructions. While existing approaches often rely on preference learning to enhance this ability, they typically overlook the relationships between the permissible response spaces of different instructions, which restricts a model to align with subtle and diverse constraint variations. To address this, we propose Cross-Relational Preference Learning (CRPL), a novel framework for constructing preference data that explicitly models inter-instruction relationships through two key techniques: Cross-Relationship Perturbation and Cross-Region Pair Sampling. This enables the generation of more diverse preference data that captures a wide spectrum of constraint variations. Additionally, we introduce an atomic constraint-based verification mechanism to rigorously assess response satisfaction, ensuring high-quality preference pair construction. Extensive experiments across multiple preference learning methods (e.g., DPO, KTO), LLM backbones and four instruction-following benchmarks demonstrate that our approach achieves substantial improvements over prior baselines and exhibits strong generalization.