mlsanityData to 7 Oct 2026

From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learning

arXiv 2604.10517 ↗First posted 12 Apr 2026cs.AIcited by 1

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Modern vision-language models achieve strong performance in static perception, but remain limited in the complex spatiotemporal reasoning required for embodied, egocentric tasks. A major source of failure is their reliance on temporal priors learned from passive video data, which often leads to spatiotemporal hallucinations and poor generalization in dynamic environments. To address this, we present EgoTSR, a curriculum-based framework for learning task-oriented spatiotemporal reasoning. EgoTSR is built on the premise that embodied reasoning should evolve from explicit spatial understanding to internalized task-state assessment and finally to long-horizon planning. To support this paradigm, we construct EgoTSR-Data, a large-scale dataset comprising 46 million samples organized into three stages: Chain-of-Thought (CoT) supervision, weakly supervised tagging, and long-horizon sequences. Extensive experiments demonstrate that EgoTSR effectively eliminates chronological biases, achieving 92.4% accuracy on long-horizon logical reasoning tasks while maintaining high fine-grained perceptual precision, significantly outperforming existing open-source and closed-source state-of-the-art models.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 54 references that are in the map. First posted on arXiv 12 Apr 2026.

Builds on

45 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation23 Jul
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 43

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2016Learning without Forgettingcited by 2,465
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  11. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  12. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
31 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  7. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  9. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2018Learning Dexterous In-Hand Manipulationcited by 643
  12. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  13. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  14. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  15. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  16. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  17. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  18. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  19. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  20. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  21. 2023Reward Design with Language Modelscited by 257
  22. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  23. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  24. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  25. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  26. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  27. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  28. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 40
  29. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  30. 2024Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?cited by 11
  31. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments
  2. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
  3. 2026Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence
  4. 2026A Progressive Training Strategy for Embodied Vision-Language Models to Mitigate Spatio-Temporal Hallucinations
  5. 2026Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learning
  6. 2025ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videos
  7. 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning
  8. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks