mlsanityData to 7 Oct 2026

ReMoT: Reinforcement Learning with Motion Contrast Triplets

arXiv 2603.00461 ↗First posted 28 Feb 2026cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

We present ReMoT, a unified training paradigm to systematically address the fundamental shortcomings of VLMs in spatio-temporal consistency -- a critical failure point in navigation, robotics, and autonomous driving. ReMoT integrates two core components: (1) A rule-based automatic framework that generates ReMoT-16K, a large-scale (16.5K triplets) motion-contrast dataset derived from video meta-annotations, surpassing costly manual or model-based generation. (2) Group Relative Policy Optimization, which we empirically validate yields optimal performance and data efficiency for learning this contrastive reasoning, far exceeding standard Supervised Fine-Tuning. We also construct the first benchmark for fine-grained motion contrast triplets to measure a VLM's discrimination of subtle motion attributes (e.g., opposing directions). The resulting model achieves state-of-the-art performance on our new benchmark and multiple standard VLM benchmarks, culminating in a remarkable 25.1% performance leap on spatio-temporal reasoning tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 89 of its 117 references that are in the map. First posted on arXiv 28 Feb 2026.

Builds on

89 of 117 references.

References · 89

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2024Mixtral of Expertscited by 1,785
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  13. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  14. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  15. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  16. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  17. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  18. 2024Yi: Open Foundation Models by 01.AIcited by 706
  19. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  20. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  21. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  22. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  23. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  24. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  25. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  26. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  27. 2024Long Context Transfer from Language to Visioncited by 502
  28. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  29. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  30. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  31. 2022Visual Language Maps for Robot Navigationcited by 362
  32. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  33. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  34. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  35. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  36. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
  37. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  38. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning
  2. 2026MoRL: Reinforced Reasoning for Unified Motion Understanding and Generation
  3. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
  4. 2026SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning
  5. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learning
  6. 2026Wan-R1: Verifiable-Reinforcement Learning for Video Reasoning
  7. 2026SA-VLA: Spatially-Aware Flow-Matching for Vision-Language-Action Reinforcement Learning
  8. 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving