# Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning

arXiv: https://arxiv.org/abs/2507.07197 · HTML: https://mlsanity.com/paper/2507.07197/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 39 of its 57 references that are in the map.
- First posted on arXiv 9 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Decoupling Representation Learning from Reinforcement Learning](https://mlsanity.com/paper/2009.08319/) (2020)
- [Masked Visual Pre-training for Motor Control](https://mlsanity.com/paper/2203.06173/) (2022)
- [Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?](https://mlsanity.com/paper/2303.18240/) (2023)
- [Continual Pre-training of Language Models](https://mlsanity.com/paper/2302.03241/) (2023)
- [Unsupervised State Representation Learning in Atari](https://mlsanity.com/paper/1906.08226/) (2019)
- [Unsupervised Learning of Object Landmarks through Conditional Image Generation](https://mlsanity.com/paper/1806.07823/) (2018)
- [Unsupervised Learning of Object Keypoints for Perception and Control](https://mlsanity.com/paper/1906.11883/) (2019)
- [Multimodal Masked Autoencoders Learn Transferable Representations](https://mlsanity.com/paper/2205.14204/) (2022)
- [Pretraining Representations for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2106.04799/) (2021)
- [RRL: Resnet as representation for Reinforcement Learning](https://mlsanity.com/paper/2107.03380/) (2021)
- [Continual Pre-Training Mitigates Forgetting in Language and Vision](https://mlsanity.com/paper/2205.09357/) (2022)
- [Mixtures of Experts Unlock Parameter Scaling for Deep RL](https://mlsanity.com/paper/2402.08609/) (2024)
- [Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes](https://mlsanity.com/paper/2211.15144/) (2022)
- [Scaling Instructable Agents Across Many Simulated Worlds](https://mlsanity.com/paper/2404.10179/) (2024)

## Built on by (most cited first, 2 of 2)

- [A Compositional Paradigm for Foundation Models: Towards Smarter Robotic Agents](https://mlsanity.com/paper/2510.18608/) (2025)
- [Combining Trained Models in Reinforcement Learning](https://mlsanity.com/paper/2605.02159/) (2026)

## Abstract (arXiv)

The recent focus and release of pre-trained models have been a key components to several advancements in many fields (e.g. Natural Language Processing and Computer Vision), as a matter of fact, pre-trained models learn disparate latent embeddings sharing insightful representations. On the other hand, Reinforcement Learning (RL) focuses on maximizing the cumulative reward obtained via agent's interaction with the environment. RL agents do not have any prior knowledge about the world, and they either learn from scratch an end-to-end mapping between the observation and action spaces or, in more recent works, are paired with monolithic and computationally expensive Foundational Models. How to effectively combine and leverage the hidden information of different pre-trained models simultaneously in RL is still an open and understudied question. In this work, we propose Weight Sharing Attention (WSA), a new architecture to combine embeddings of multiple pre-trained models to shape an enriched state representation, balancing the tradeoff between efficiency and performance. We run an extensive comparison between several combination modes showing that WSA obtains comparable performance on multiple Atari games compared to end-to-end models. Furthermore, we study the generalization capabilities of this approach and analyze how scaling the number of models influences agents' performance during and after training.