mlsanityData to 7 Oct 2026

Aligning Compound AI Systems via System-level DPO

arXiv 2502.17721 ↗First posted 24 Feb 2025cs.LG cs.AI cs.MAcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Compound AI systems, comprising multiple interacting components such as LLMs, foundation models, and external tools, have demonstrated remarkable improvements compared to single models in various tasks. To ensure their effective deployment in real-world applications, aligning these systems with human preferences is crucial. However, aligning the compound system via policy optimization, unlike the alignment of a single model, is challenging for two main reasons: (i) non-differentiable interactions between components make end-to-end gradient-based optimization method inapplicable, and (ii) system-level preferences cannot be directly transformed into component-level preferences. To address these challenges, we first formulate compound AI systems as Directed Acyclic Graphs (DAGs), explicitly modeling both component interactions and the associated data flows. Building on this formulation, we introduce SysDPO, a framework that extends Direct Preference Optimization (DPO) to enable joint system-level alignment. We propose two variants, SysDPO-Direct and SysDPO-Sampling, tailored for scenarios depending on whether we construct a system-specific preference dataset. We empirically demonstrate the effectiveness of our approach across two applications: the joint alignment of a language model and a diffusion model, and the joint alignment of an LLM collaboration system.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 53 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

30 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Learning Dynamics of LLM Finetuning15 Jul
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 28

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  12. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
16 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  3. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  4. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  5. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  6. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  7. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  8. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  9. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  10. 2023Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debatecited by 94
  11. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  12. 2017Automated Scalable Bayesian Inference via Hilbert Coresetscited by 55
  13. 2021Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigationcited by 53
  14. 2024DiffusionAgent: Navigating Expert Models for Agentic Image Generationcited by 27
  15. 2015Tight Bounds for Approximate Carath'eodory and Beyondcited by 8
  16. 2024A Blueprint Architecture of Compound AI Systems for Enterprisecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directionscited by 4
  2. 2025LLM Chemistry Estimation for Multi-LLM Recommendationcited by 2
  3. 2026Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systemscited by 1
  4. 2026Do Modules Stay in Their Lane? Role Drift in Compound LLM Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewards
  2. 2025C2-DPO: Constrained Controlled Direct Preference Optimization
  3. 2025Optimizing Model Selection for Compound AI Systems
  4. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  5. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
  6. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  7. 2025Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?
  8. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models