Integrating Vision Foundation Models with Reinforcement Learning for Enhanced Object Interaction
On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation
Abstract (arXiv)
This paper presents a novel approach that integrates vision foundation models with reinforcement learning to enhance object interaction capabilities in simulated environments. By combining the Segment Anything Model (SAM) and YOLOv5 with a Proximal Policy Optimization (PPO) agent operating in the AI2-THOR simulation environment, we enable the agent to perceive and interact with objects more effectively. Our comprehensive experiments, conducted across four diverse indoor kitchen settings, demonstrate significant improvements in object interaction success rates and navigation efficiency compared to a baseline agent without advanced perception. The results show a 68% increase in average cumulative reward, a 52.5% improvement in object interaction success rate, and a 33% increase in navigation efficiency. These findings highlight the potential of integrating foundation models with reinforcement learning for complex robotic tasks, paving the way for more sophisticated and capable autonomous agents.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 22 of its 33 references that are in the map. First posted on arXiv 7 Aug 2025.
Builds on
22 of 33 references.
References · 22
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
- 2023Segment Anythingcited by 8,234
- 2017Mask R-CNNcited by 7,277
- 2016Feature Pyramid Networks for Object Detectioncited by 5,036
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2015Trust Region Policy Optimizationcited by 2,580
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2016YOLO9000: Better, Faster, Strongercited by 1,431
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
- 2015End-to-End Training of Deep Visuomotor Policiescited by 875
10 more
- 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
- 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
- 2018Learning Dexterous In-Hand Manipulationcited by 687
- 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
- 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
- 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
- 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
- 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
- 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 165
- 2018Visual Representations for Semantic Target Driven Navigationcited by 89
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Learning Interactive World Model for Object-Centric Reinforcement Learning
- 2026Gym-V: A Unified Vision Environment System for Agentic Vision Research
- 2025SemNav: A Model-Based Planner for Zero-Shot Object Goal Navigation Using Vision-Foundation Models
- 2026Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization
- 2025Reinforcement Learning for Large Model: A Survey
- 2025HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agents
- 2023Herd's Eye View: Improving Game AI Agent Learning with Collaborative Perception
- 2023Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Own