# COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams

arXiv: https://arxiv.org/abs/2604.17693 · HTML: https://mlsanity.com/paper/2604.17693/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [FACMAC: Factored Multi-Agent Centralised Policy Gradients](https://mlsanity.com/paper/2003.06709/) (2020)
- [LLM Multi-Agent Systems: Challenges and Open Problems](https://mlsanity.com/paper/2402.03578/) (2024)
- [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016)
- [Off-Policy Multi-Agent Decomposed Policy Gradients](https://mlsanity.com/paper/2007.12322/) (2020)
- [Hindsight Credit Assignment for Long-Horizon LLM Agents](https://mlsanity.com/paper/2603.08754/) (2026)
- [Settling the Variance of Multi-Agent Policy Gradients](https://mlsanity.com/paper/2108.08612/) (2021)
- [Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation](https://mlsanity.com/paper/2511.02303/) (2025)
- [Counterfactual Credit Policy Optimization for Multi-Agent Collaboration](https://mlsanity.com/paper/2603.21563/) (2026)
- [Difference Rewards Policy Gradients](https://mlsanity.com/paper/2012.11258/) (2020)
- [Counterfactual Effect Decomposition in Multi-Agent Sequential Decision Making](https://mlsanity.com/paper/2410.12539/) (2024)
- [Sequential Multi-Agent Dynamic Algorithm Configuration](https://mlsanity.com/paper/2510.23535/) (2025)

## Built on by (most cited first, 1 of 1)

- [EvoSteer: Online Self-Evolving Graph Orchestration via Reference-Anchored Credit Assignment](https://mlsanity.com/paper/2609.38661/) (2026)

## Abstract (arXiv)

In cooperative teams where agents act in a fixed order and share a single team-level reward (multi-agent language systems, sequential robotic tasks), per-agent credit assignment is under-determined. Critic-based approaches scale poorly as the number of agents grows owing to the costly maintenance of joint/factored critic(s), whereas the existing critic-free alternatives have other issues: common credit across agents that couples every agent's signal to teammate noise, importance-sampling corrections for upstream-update staleness that incur variance exponential in team size, or per-agent counterfactual replay that isolates each agent's effect at the price of extra environment or reward calls. We propose COSAC, a critic-free per-agent policy gradient for sequential cooperative teams. COSAC fits an additive per-agent decomposition of the team reward by a single ridge regression on the rollout batch (giving each agent a learning signal decoupled from teammate noise), and computes each agent's counterfactual advantage from fictitious continuations of the current policy (policy forward passes that replace both importance-sampling reweighting and per-agent environment replay, at no extra environment or reward cost). The estimator instantiates the Sequential Aristocrat Utility (SeqAU), our extension of Wolpert and Tumer's (2001) aristocrat utility to sequential teams. We prove bias and variance bounds on SeqAU credits that stay controlled as the team grows. Our controlled study on sequential bandits demonstrates that COSAC attains the lowest advantage MSE and consistently low learning regret across team sizes up to K = 16. On the AI2 Reasoning Challenge (ARC) task, where four Qwen3-0.6B agents reason in turn about a grade-school science question, COSAC attains faster convergence than the other critic-free baselines.