mlsanityData to 14 Oct 2026

CLSP: High-Fidelity Contrastive Language-State Pre-training for Agent State Representation

arXiv 2409.15806 ↗First posted 24 Sep 2024cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

With the rapid development of artificial intelligence, multimodal learning has become an important research area. For intelligent agents, the state is a crucial modality to convey precise information alongside common modalities like images, videos, and language. This becomes especially clear with the broad adoption of reinforcement learning and multimodal large language models. Nevertheless, the representation of state modality still lags in development. To this end, we propose a High-Fidelity Contrastive Language-State Pre-training (CLSP) method, which can accurately encode state information into general representations for both reinforcement learning and multimodal large language models. Specifically, we first design a pre-training task based on the classification to train an encoder with coarse-grained information. Next, we construct data pairs of states and language descriptions, utilizing the pre-trained encoder to initialize the CLSP encoder. Then, we deploy contrastive learning to train the CLSP encoder to effectively represent precise state information. Additionally, we enhance the representation of numerical information using the Random Fourier Features (RFF) method for high-fidelity mapping. Extensive experiments demonstrate the superior precision and generalization capabilities of our representation, achieving outstanding results in text-state retrieval, reinforcement learning navigation tasks, and multimodal large language model understanding.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 31 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

24 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policy28 Jun
  2. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  9. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  12. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
10 more
  1. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  3. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  4. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  5. 2023GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localizationcited by 187
  6. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 179
  7. 2022CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classificationcited by 65
  8. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  9. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  10. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Training
  2. 2026CPPO: Contrastive Perception Policy Optimization for VLM Agents
  3. 2024LLM-Empowered State Representation for Reinforcement Learning
  4. 2024MORE-3S:Multimodal-based Offline Reinforcement Learning with Shared Semantic Spaces
  5. 2025CLIP-RL: Aligning Language and Policy Representations for Task Transfer in Reinforcement Learning
  6. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  7. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
  8. 2026Contrastive Reinforced Policy Optimization via Privileged Self-Distillation