# Learning Robot Policies from Sparse Success Signals via STL-Guided Stein Variational Policy Gradient

arXiv: https://arxiv.org/abs/2609.31606 · HTML: https://mlsanity.com/paper/2609.31606/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning](https://mlsanity.com/paper/2010.03950/) (2020)
- [Reinforcement Learning With Temporal Logic Rewards](https://mlsanity.com/paper/1612.03471/) (2016)
- [LTL2Action: Generalizing LTL Instructions for Multi-Task RL](https://mlsanity.com/paper/2102.06858/) (2021)
- [Backpropagation through Signal Temporal Logic Specifications: Infusing Logical Structure into Gradient-Based Methods](https://mlsanity.com/paper/2008.00097/) (2020)
- [Stein Variational Policy Gradient](https://mlsanity.com/paper/1704.02399/) (2017)
- [Signal Temporal Logic Neural Predictive Control](https://mlsanity.com/paper/2309.05131/) (2023)
- [Funnel-based Reward Shaping for Signal Temporal Logic Tasks in Reinforcement Learning](https://mlsanity.com/paper/2212.03181/) (2022)
- [STLCG++: A Masking Approach for Differentiable Signal Temporal Logic Specification](https://mlsanity.com/paper/2501.04194/) (2025)
- [TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks](https://mlsanity.com/paper/2510.00225/) (2025)
- [Disentangled Planning and Control in Vision Based Robotics via Reward Machines](https://mlsanity.com/paper/2012.14464/) (2020)
- [Prescribed Performance Control Guided Policy Improvement for Satisfying Signal Temporal Logic Tasks](https://mlsanity.com/paper/1903.04340/) (2019)
- [Agile Flight Emerges from Multi-Agent Competitive Racing](https://mlsanity.com/paper/2512.11781/) (2025)
- [STL-SVPIO: Signal Temporal Logic guided Stein Variational Path Integral Optimization](https://mlsanity.com/paper/2603.13333/) (2026)
- [Reinforcement Learning for Flow-Matching Policies with Density Transport](https://mlsanity.com/paper/2606.08602/) (2026)

## Abstract (arXiv)

Learning robot policies for tasks with sparse success signals is challenging when completion depends on coordinated actions, precise contact outcomes, or satisfying several conditions together. Intricate physical interactions with the world further complicate these requirements. Prior work using conventional reward shaping mechanisms provides dense feedback but local progress might not translate into eventual task completion. We present Signal Temporal Logic-guided Stein Variational Policy Gradient (STL-SVPG), a population-based method that uses smooth STL robustness as a trajectory-level training objective. Differentiating this objective through the dynamics assigns credit to policy actions according to their effect on the complete task specification, rather than local progress alone. We evaluate the approach on six quadcopter and manipulator tasks that involves event-triggered responses, strictly ordered behavior, responses within specified deadlines, and physical interaction with the world. STL-SVPG achieves the highest mean success rate among the compared methods on five of six benchmarks. Simulation-trained policies trained in simulation transfer temporal and contact task behavior to the real world.