Squint: Fast Visual Reinforcement Learning for Sim-to-Real Robotics
Not on the map. Why?
Abstract (arXiv)
Visual reinforcement learning is appealing for robotics but expensive. Off-policy methods are sample-efficient yet slow while on-policy methods parallelize well but waste samples. Recent work has shown that off-policy methods can train faster than on-policy methods in wall-clock time for state-based control. Extending this to vision remains challenging, where high-dimensional input images complicate training dynamics and introduce substantial storage and encoding overhead. To address these challenges, we introduce Squint, a visual Soft Actor Critic method that achieves faster wall-clock training than prior visual off-policy and on-policy methods. Squint achieves this via parallel simulation, a distributional critic, resolution squinting, layer normalization, a tuned update-to-data ratio, and an optimized implementation. We evaluate on the SO-101 Task Set, a new suite of eight manipulation tasks in ManiSkill3 with heavy domain randomization, and demonstrate sim-to-real transfer to a real SO-101 robot. We train policies for 15 minutes on a single RTX 3090 GPU, with most tasks converging in under 6 minutes.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 66 of its 90 references that are in the map. First posted on arXiv 24 Feb 2026.
Builds on
66 of 90 references.
References · 66
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2016Layer Normalizationcited by 5,579
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
- 2023Mastering Diverse Domains through World Modelscited by 1,193
- 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
- 2015Prioritized Experience Replaycited by 1,070
38 more
- 2018DeepMind Control Suitecited by 935
- 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
- 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2017A Distributional Perspective on Reinforcement Learningcited by 743
- 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
- 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
- 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
- 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
- 2020Reinforcement Learning with Augmented Datacited by 452
- 2019Learning agile and dynamic motor skills for legged robotscited by 402
- 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
- 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
- 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
- 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
- 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
- 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
- 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
- 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 159
- 2023Robot Parkour Learningcited by 115
- 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
- 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
- 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
- 2018GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learningcited by 77
- 2024Simplifying Deep Temporal Difference Learningcited by 71
- 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
- 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
- 2025MuJoCo Playgroundcited by 57
- 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
- 2024RLDG: Robotic Generalist Policy Distillation via Reinforcement Learningcited by 42
- 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 42
- 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
- 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
- 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
- 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 32
- 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
- 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
50 most cited of 66 All 66 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.