mlsanityData to 7 Oct 2026

Online Joint Fine-tuning of Multi-Agent Flows

arXiv 2406.04516 ↗First posted 6 Jun 2024cs.LGcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

A Flow is a collection of component models ("Agents") which constructs the solution to a complex problem via iterative communication. Flows have emerged as state of the art architectures for code generation, and are the raison d'etre for frameworks like Autogen. However, flows are currently constructed via a combination of manual prompt engineering and stagewise supervised learning techniques; the latter is limited to acyclic flows with granular node supervision. In this writeup I describe a procedure for online joint fine-tuning of an entire flow inspired by the Learning to Search framework. The approach leverages simulator access to reduce preferences over entire episodes to preferences over individual node outputs; when the components are language models the latter is a well-studied problem. The approach is applicable to reward-free settings (e.g., text feedback) if an episode evaluator model is available. I apply to the multi-hop QA dataset Musique achieving a state-of-the-art result.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 13 of its 26 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

13 of 26 references.

References · 13

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2023Mistral 7Bcited by 3,065
  6. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  7. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  8. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  9. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  10. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  11. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  12. 2024Learn Your Reference Model for Real Good Alignmentcited by 56
1 more
  1. 2023Bridging RL Theory and Practice with the Effective Horizoncited by 31

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 69
  2. 2024Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learningcited by 11

Similar, not cited

Close in wording, no citation either way

  1. 2026MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimization
  2. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
  3. 2025ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization
  4. 2026Reinforced Collaboration in Multi-Agent Flow Networks
  5. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Use
  6. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
  7. 2025Fine-tuning Flow Matching Generative Models with Intermediate Feedback
  8. 2025Don't Just Fine-tune the Agent, Tune the Environment