# Embodiment-Induced Coordination Regimes in Tabular Multi-Agent Q-Learning

arXiv: https://arxiv.org/abs/2601.17454 · HTML: https://mlsanity.com/paper/2601.17454/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 22 references that are in the map.
- First posted on arXiv 24 Jan 2026.

## Builds on (most cited first, 4 of 4)

- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems](https://mlsanity.com/paper/1709.08071/) (2017)
- [A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity](https://mlsanity.com/paper/1707.09183/) (2017)
- [Leveraging human knowledge in tabular reinforcement learning: A study of human subjects](https://mlsanity.com/paper/1805.05769/) (2018)

## Abstract (arXiv)

Centralized value learning underlies a broad class of multi-agent reinforcement learning methods, but its claimed advantage is typically evaluated in settings that confound coordination structure with function approximation and partial observability. We isolate coordination structure in a fully tabular 8x8 predator-prey gridworld with explicit speed and stamina constraints, comparing all four pairings of Independent and Centralized Q-Learning across three kinematic regimes over 10 seeds. Fully independent learning (IQL-IQL) yields shorter episodes and higher predator returns than the fully centralized configuration (CQL-CQL) in every regime and every seed (Wilcoxon p = 0.00195, Cliff's delta = 1.0). Asymmetric IQL-CQL pairings produce coordination breakdowns that persist across the 40,000 episode training budget rather than resolving as transient instability. A best-response test against frozen IQL-IQL predators shows that even the strongest configuration has not converged to equilibrium at 40k episodes, so between-configuration differences reflect learning dynamics under a shared budget rather than end-state performance. We propose a mechanism we call temporal synchronization lock: a shared value function couples all agent decisions, so when one agent is stamina-limited the joint Q-value forces capable partners into suboptimal waits, while independent learners continue asynchronous pursuit. Because this pathology arises from credit assignment rather than function approximation, we conjecture analogous effects may arise in deep MARL methods that centralize credit during training. Centralized coordination is not uniformly beneficial; its advantage over independent learning shrinks or disappears under embodiment constraints, and mixed centralized-independent pairings can perform worse than either uniform choice.