mlsanityData to 14 Oct 2026

MORAL: A Multimodal Reinforcement Learning Framework for Decision Making in Autonomous Laboratories

arXiv 2504.03153 ↗First posted 4 Apr 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

We propose MORAL (a multimodal reinforcement learning framework for decision making in autonomous laboratories) that enhances sequential decision-making in autonomous robotic laboratories through the integration of visual and textual inputs. Using the BridgeData V2 dataset, we generate fine-tuned image captions with a pretrained BLIP-2 vision-language model and combine them with visual features through an early fusion strategy. The fused representations are processed using Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) agents. Experimental results demonstrate that multimodal agents achieve a 20% improvement in task completion rates and significantly outperform visual-only and textual-only baselines after sufficient training. Compared to transformer-based and recurrent multimodal RL models, our approach achieves superior performance in cumulative reward and caption quality metrics (BLEU, METEOR, ROUGE-L). These results highlight the impact of semantically aligned language cues in enhancing agent learning efficiency and generalization. The proposed framework contributes to the advancement of multimodal reinforcement learning and embodied AI systems in dynamic, real-world environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 6 of its 8 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

6 of 8 references.

References · 6

  1. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  2. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  3. 2017Deep Reinforcement Learning-based Image Captioning with Embedding Rewardcited by 40
  4. 2022Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysiscited by 22
  5. 2018Image Captioning based on Deep Reinforcement Learningcited by 3
  6. 2024Enhancing Image Caption Generation Using Reinforcement Learning with Human Feedbackcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026IAPO: Input Attribution-Aware Policy Optimization for Tool Use in Small Multimodal Agents
  2. 2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learning
  3. 2026MA-VLCM: A Vision Language Critic Model for Value Estimation of Policies in Multi-Agent Team Settings
  4. 2024Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models
  5. 2025PRIMT: Preference-based Reinforcement Learning with Multimodal Feedback and Trajectory Synthesis from Foundation Models
  6. 2025DeepEyesV2: Toward Agentic Multimodal Model
  7. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
  8. 2026Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models