mlsanityData to 7 Oct 2026

GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training

arXiv 2512.13043 ↗First posted 15 Dec 2025cs.CV cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Multi-turn reinforcement learning (RL) for multi-modal agents built upon vision-language models (VLMs) is hampered by sparse rewards and long-horizon credit assignment. Recent methods densify the reward by querying a teacher that provides step-level feedback, e.g., Guided Thought Reinforcement (GTR) and On-Policy Distillation, but rely on costly, often privileged models as the teacher, limiting practicality and reproducibility. We introduce GTR-Turbo, a highly efficient upgrade to GTR that matches its performance without training on or querying an expensive teacher model. Specifically, GTR-Turbo merges the weights of checkpoints produced during ongoing RL training and then uses the resulting merged model as a "free" teacher to guide subsequent RL via supervised fine-tuning or soft logit distillation. This design removes dependence on privileged VLMs (e.g., GPT or Gemini), mitigates the "entropy collapse" observed in prior work, and maintains stable training. Across diverse visual agentic tasks, GTR-Turbo improves the accuracy of the baseline model by 10-30% while reducing wall-clock training time by 50% and compute cost by 60% relative to GTR.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 72 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

61 of 72 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents19 Oct
  2. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation28 Sep
  3. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May
  4. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedback12 Oct
  5. 2023Physically Grounded Vision-Language Models for Robotic Manipulation5 Sep
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  7. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May
  8. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  9. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agents29 Jan

Other references · 52

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  12. 2025Qwen3-VL Technical Reportcited by 2,534
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  4. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  7. 2022Editing Models with Task Arithmeticcited by 1,076
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  12. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  13. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  14. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  15. 2025Process Reinforcement through Implicit Rewardscited by 478
  16. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  17. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  18. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  19. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  20. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  21. 2015Policy Distillationcited by 414
  22. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  23. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  24. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  25. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  26. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  27. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  28. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  29. 2024Free Process Rewards without Process Labelscited by 162
  30. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  31. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  32. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  33. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 60
  34. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 58
  35. 2024LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedbackcited by 40
  36. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 36
  37. 2025Model Merging in Pre-training of Large Language Modelscited by 36
  38. 2024WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agentscited by 24

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  2. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  3. 2026Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Code as Reward: Empowering Reinforcement Learning with VLMs
  2. 2025G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning
  3. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
  4. 2026CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving
  5. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learning
  6. 2026Selective Transfer of RL Updates for Visual Reasoning
  7. 2025Refined Policy Distillation: From VLA Generalists to RL Experts
  8. 2025Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs