mlsanityData to 14 Oct 2026

Triplet2Track: A Hierarchical System with Object-Centric Representations for Reliable Long-Horizon Manipulation

arXiv 2608.22800 ↗First posted 24 Aug 2026cs.RO cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Ensuring reliability in uncertain environments remains difficult for long-horizon robotic manipulation. End-to-end VLA models are data-heavy and opaque, making diagnosis and verification difficult. Hierarchical pipelines are more interpretable, but their plans are often weakly grounded in observations, weakly aligned with low-level actions, and computed without online feedback, leading to open-loop behavior and hallucinations. To address these issues, we introduce the Triplet-to-Track System (TTS), a closed-loop long-horizon imitation learning system that uses human videos to reduce reliance on robot-collected data. TTS represents high-level subgoals as instance-grounded triplets, translates them into continuous track priors for execution, and monitors task progress from observations for online replanning. Across diverse real-world long-horizon tasks, TTS achieves a 74.8% average success rate and supports object-level and compositional generalization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 48 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

44 of 48 references.

References · 44

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  3. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  7. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  10. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  12. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
32 more
  1. 2018Behavioral Cloning from Observationcited by 492
  2. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  3. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  4. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  5. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  6. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  7. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  8. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
  9. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  10. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  11. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 175
  12. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 164
  13. 2024Flow as the Cross-Domain Manipulation Interfacecited by 132
  14. 2024CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videoscited by 128
  15. 2022VideoDex: Learning Dexterity from Internet Videoscited by 106
  16. 2023ISR-LLM: Iterative Self-Refined Large Language Model for Long-Horizon Sequential Task Planningcited by 77
  17. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75
  18. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 51
  19. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulationcited by 48
  20. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulationcited by 45
  21. 2025Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulationcited by 42
  22. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39
  23. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 39
  24. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 37
  25. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 36
  26. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 26
  27. 2024Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitationcited by 18
  28. 2024MLDT: Multi-Level Decomposition for Complex Long-Horizon Robotic Task Planning with Open-Source Large Language Modelcited by 15
  29. 2024Can Transformers Capture Spatial Relations between Objects?cited by 11
  30. 2024ARCH: Hierarchical Hybrid Learning for Long-Horizon Contact-Rich Robotic Assemblycited by 10
  31. 2024What's the Move? Hybrid Imitation Learning via Salient Pointscited by 9
  32. 2025STEP Planner: Constructing cross-hierarchical subgoal tree as an embodied long-horizon task plannercited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation
  2. 2025From Watch to Imagine: Steering Long-horizon Manipulation via Human Demonstration and Future Envisionment
  3. 2025MIND-V: Hierarchical World Model for Long-Horizon Robotic Manipulation with RL-based Physical Alignment
  4. 2026ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks ↗
  5. 2023LHManip: A Dataset for Long-Horizon Language-Grounded Manipulation Tasks in Cluttered Tabletop Environments
  6. 2026SADP: Subgoal-Aware Diffusion Policy for Long-Horizon Manipulation Learned from Foundation Model Generated Demonstrations
  7. 2026Beyond the Majority: Long-tail Imitation Learning for Robotic Manipulation
  8. 2025ManiLong-Shot: Interaction-Aware One-Shot Imitation Learning for Long-Horizon Manipulation