mlsanityData to 14 Oct 2026

Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning

arXiv 2609.37519 ↗First posted 29 Sep 2026cs.RO cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Video-based policy learning is particularly promising, as it illustrates target behaviors without requiring action annotations or embodiment-matched demonstrations. A central challenge is deciding what information should be transferred from the video to the robot. Existing approaches commonly convert visual observations into scalar similarity or value signals, or ask foundation models to directly generate reward code. These approaches can make the temporal structure of a task difficult to inspect, ground, and reuse. We present Video2STL, a framework that converts observation-only videos into parametric Signal Temporal Logic (STL) specifications and uses the resulting formal representation for robot learning. A vision-language model extracts an embodiment-independent semantic event trace and constructs a bank of symbolic temporal specifications. The model determines the task structure, while numerical predicate thresholds and temporal bounds are grounded from successful robot trajectories. For policy learning, we separate short- and long-timescale temporal information: short-horizon specifications provide dense rewards through rolling-window quantitative robustness, while a causal monitor over a retained long-horizon specification provides one-time progress rewards for valid temporal prefixes. The same representation supports cross-embodiment transfer from human or animal videos to robot control. Across four manipulation tasks, Video2STL achieves 85.8% average success-once and 67.0% success-at-end, compared with 81.5%/59.5% for native dense PPO and 65.0%/42.3% for Text2Reward; in quadruped locomotion, Qwen-3.8 and GPT-5.6-based Video2STL policies achieve 100% success across velocities from 0.3 to 2.1,m/s while remaining competitive in high-speed energy efficiency. Project webpage: hrefhttps://video2stl.github.io/video2stl.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

25 of 37 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  3. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  4. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  6. 2023Reward Design with Language Modelscited by 261
  7. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  8. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  9. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 165
  10. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 160
  11. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  12. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 148
13 more
  1. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  2. 2016Reinforcement Learning With Temporal Logic Rewardscited by 117
  3. 2024Vision Language Models are In-Context Value Learnerscited by 90
  4. 2023Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environmentscited by 53
  5. 2023Barkour: Benchmarking Animal-level Agility with Quadruped Robotscited by 41
  6. 2020Tractable Reinforcement Learning of Signal Temporal Logic Objectivescited by 22
  7. 2020Model-based Reinforcement Learning from Signal Temporal Logic Specificationscited by 18
  8. 2021Learning from Demonstrations using Signal Temporal Logiccited by 17
  9. 2024Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learningcited by 9
  10. 2024Robust-Locomotion-by-Logic: Perturbation-Resilient Bipedal Locomotion via Signal Temporal Logic Guided Model Predictive Controlcited by 8
  11. 2021Learning Quadruped Locomotion Policies using Logical Rulescited by 6
  12. 2024Learning to Adapt through Bio-Inspired Gait Strategies for Versatile Quadruped Locomotioncited by 6
  13. 2026Learning Gait-Aware Quadruped Locomotion with Temporal Logic Specificationscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning
  2. 2026G0.5: One Autoregressive Stream for Robot Reasoning and Action
  3. 2026STeP: Signal Temporal Logic for Precise Specifications for Action Generation with Vision Language Models
  4. 2026Native Video-Action Pretraining for Generalizable Robot Control
  5. 2025ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context
  6. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
  7. 2026DLAM: Distributional Latent Actions with Temporal Constraints
  8. 2026Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models