mlsanityData to 14 Oct 2026

VIRAL: Vision-grounded Integration for Reward design And Learning

arXiv 2505.22092 ↗First posted 28 May 2025cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The alignment between humans and machines is a critical challenge in artificial intelligence today. Reinforcement learning, which aims to maximize a reward function, is particularly vulnerable to the risks associated with poorly designed reward functions. Recent advancements has shown that Large Language Models (LLMs) for reward generation can outperform human performance in this context. We introduce VIRAL, a pipeline for generating and refining reward functions through the use of multi-modal LLMs. VIRAL autonomously creates and interactively improves reward functions based on a given environment and a goal prompt or annotated image. The refinement process can incorporate human feedback or be guided by a description generated by a video LLM, which explains the agent's policy in video form. We evaluated VIRAL in five Gymnasium environments, demonstrating that it accelerates the learning of new behaviors while ensuring improved alignment with user intent. The source-code and demo video are available at: https://github.com/VIRAL-UCBL1/VIRAL and https://youtu.be/Hqo82CxVT38.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 27 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

21 of 27 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Models19 Oct
  2. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learning20 Sep
  3. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  5. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  10. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  11. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  12. 2023Reward Design with Language Modelscited by 261
3 more
  1. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 159
  2. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  3. 2023Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Roboticscited by 36

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policiescited by 8

Similar, not cited

Close in wording, no citation either way

  1. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
  2. 2024Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos
  3. 2025Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning
  4. 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignment
  5. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs
  6. 2025Leveraging LLMs for reward function design in reinforcement learning control tasks
  7. 2025Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shaping
  8. 2025Reward Evolution with Graph-of-Thoughts: A Bi-Level Language Model Framework for Reinforcement Learning