ReMoT: Reinforcement Learning with Motion Contrast Triplets
Not on the map. Why?
Abstract (arXiv)
We present ReMoT, a unified training paradigm to systematically address the fundamental shortcomings of VLMs in spatio-temporal consistency -- a critical failure point in navigation, robotics, and autonomous driving. ReMoT integrates two core components: (1) A rule-based automatic framework that generates ReMoT-16K, a large-scale (16.5K triplets) motion-contrast dataset derived from video meta-annotations, surpassing costly manual or model-based generation. (2) Group Relative Policy Optimization, which we empirically validate yields optimal performance and data efficiency for learning this contrastive reasoning, far exceeding standard Supervised Fine-Tuning. We also construct the first benchmark for fine-grained motion contrast triplets to measure a VLM's discrimination of subtle motion attributes (e.g., opposing directions). The resulting model achieves state-of-the-art performance on our new benchmark and multiple standard VLM benchmarks, culminating in a remarkable 25.1% performance leap on spatio-temporal reasoning tasks.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 89 of its 117 references that are in the map. First posted on arXiv 28 Feb 2026.
Builds on
89 of 117 references.
References · 89
- 2023GPT-4 Technical Reportcited by 18,132
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2023Visual Instruction Tuningcited by 7,792
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024GPT-4o System Cardcited by 4,467
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2023Qwen Technical Reportcited by 3,273
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
38 more
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2024Mixtral of Expertscited by 1,785
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
- 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
- 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
- 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
- 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
- 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
- 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
- 2024Yi: Open Foundation Models by 01.AIcited by 706
- 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
- 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
- 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
- 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
- 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
- 2024Long Context Transfer from Language to Visioncited by 502
- 2023Generative Multimodal Models are In-Context Learnerscited by 446
- 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
- 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
- 2022Visual Language Maps for Robot Navigationcited by 362
- 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
- 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
- 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
- 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
- 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
- 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
- 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
50 most cited of 89 All 89 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning
- 2026MoRL: Reinforced Reasoning for Unified Motion Understanding and Generation
- 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
- 2026SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning
- 2025One RL to See Them All: Visual Triple Unified Reinforcement Learning
- 2026Wan-R1: Verifiable-Reinforcement Learning for Video Reasoning
- 2026SA-VLA: Spatially-Aware Flow-Matching for Vision-Language-Action Reinforcement Learning
- 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving