mlsanityData to 14 Oct 2026

Algorithm-Relative Trajectory Valuation in Policy Gradient Control

arXiv 2511.07878 ↗First posted 11 Nov 2025cs.LG cs.SY eess.SYcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We study how trajectory value depends on the learning algorithm in policy-gradient control. Using Trajectory Shapley in an uncertain LQR, we find a negative correlation between Persistence of Excitation (PE) and marginal value under vanilla REINFORCE (r≈-0.38). We prove a variance-mediated mechanism: (i) for fixed energy, higher PE yields lower gradient variance; (ii) near saddles, higher variance increases escape probability, raising marginal contribution. When stabilized (state whitening or Fisher preconditioning), this variance channel is neutralized and information content dominates, flipping the correlation positive (r≈+0.29). Hence, trajectory value is algorithm-relative. Experiments validate the mechanism and show decision-aligned scores (Leave-One-Out) complement Shapley for pruning, while Shapley identifies toxic subsets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 11 of its 22 references that are in the map. First posted on arXiv 11 Nov 2025.

Builds on

11 of 22 references.

References · 11

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  3. 2015Prioritized Experience Replaycited by 1,099
  4. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  5. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  6. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  7. 2021Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learningcited by 165
  8. 2024Capturing the Temporal Dependence of Training Data Influencecited by 35
  9. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  10. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
  11. 2024Linear Convergence of Independent Natural Policy Gradient in Games with Entropy Regularizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Rate
  2. 2024Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States
  3. 2026Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator
  4. 2025Improving Value Estimation Critically Enhances Vanilla Policy Gradient
  5. 2026Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods
  6. 2024Behind the Myth of Exploration in Policy Gradients
  7. 2024Mollification Effects of Policy Gradient Methods
  8. 2026Does "Do Differentiable Simulators Give Better Policy Gradients?'' Give Better Policy Gradients?