# Taylor Representations for Model-Free RL in Networked MDPs

arXiv: https://arxiv.org/abs/2610.05456 · HTML: https://mlsanity.com/paper/2610.05456/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 1 of its 53 references that are in the map.
- First posted on arXiv 4 Oct 2026.

## Builds on (most cited first, 1 of 1)

- [Scalable Policy Optimization for Networked Multi-Agent Reinforcement Learning with Continuous State-Action Spaces](https://mlsanity.com/paper/2607.18554/) (2026)

## Abstract (arXiv)

In Networked Markov Decision Processes, transition dynamics are often unknown and the state--action space grows rapidly with the number of agents. In this setting, Taylor representations naturally approximate Q-functions, but a naive order-n expansion over N agents requires Theta(Nn) coefficients. We justify these expansions under smooth expected future local rewards with controlled derivatives. Under this condition, finite-speed information propagation and discounting imply that local-critic Taylor coefficients decay exponentially with the graph distance to the farthest agent involved. Discarding distant-agent coefficients and marginalizing then yield scalable local Taylor representations with a bound controlled by graph locality. Building on these representations, we propose a scalable model-free actor--critic algorithm, establishing finite-sample critic and near-stationarity guarantees for a linear LSTD critic. We then introduce a more expressive neural TD parameterization. Unlike prior constructive spectral methods, our approach covers settings without access to a known local dynamics map, such as hidden switched linear--quadratic regulation. Across three control benchmarks, our method matches or outperforms spectral baselines while scaling efficiently to large graphs.