mlsanityData to 7 Oct 2026

PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control

arXiv 2608.24115 ↗First posted 25 Aug 2026cs.RO cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) can integrate long visual histories and infer behavior from a few examples, yet vision-language-action models rarely use this capacity as episode memory. Instead of a purpose-built memory module, PONDERPOUNCE reuses an MLLM's native causal context. PONDER, a pretrained System 2 MLLM, integrates episode history and demonstrations to produce continuous cognition. POUNCE, a System 1 action model, asynchronously conditions control on the newest cognition and its age. Both are jointly trained end to end without separate bridge pretraining. Optimized per-call inference on an H100 achieves p50 latencies of 78 ms for cognition-only refresh and 25 ms for action-model invocation. On RoboMME, PONDERPOUNCE achieves 60.83% success at the base data scale and 75.54% with 9x data, compared with 44.51% and 57.88% for FrameSamp+Modul. At base scale, scaling PONDER from 0.8B to 9B adds 6.71 percentage points with the POUNCE architecture unchanged. A separately trained 9B PONDER without execution history achieves only 26.21% under matched supervision. PONDERPOUNCE also achieves 12.5% success on RoboCasa-DC and demonstrates real-world applicability on four tasks under asynchronous execution, with 60.98% mean success versus 40.67% for FrameSamp+Modul.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 53 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

48 of 53 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control13 Feb
  2. 2026ST4VLA: Spatially Guided Training for Vision-Language-Action Models10 Feb
  3. 2025Asynchronous Fast-Slow Vision-Language-Action Policies for Whole-Body Robotic Manipulation23 Dec
  4. 2025Training-free Generation of Temporally Consistent Rewards from VLMs7 Jul
  5. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control9 Feb
  6. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers12 Sep
  7. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models21 Nov

Other references · 41

  1. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  2. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  3. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  4. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  6. 2024Genie: Generative Interactive Environmentscited by 706
  7. 2024Latent Action Pretraining from Videoscited by 302
  8. 2017One-Shot Imitation Learningcited by 232
  9. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  10. 2024RT-H: Action Hierarchies Using Languagecited by 204
  11. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  12. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
29 more
  1. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  2. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  3. 2025OpenHelix: A Short Survey, Empirical Analysis, and Open-Source Dual-System VLA Model for Robotic Manipulationcited by 79
  4. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  5. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  6. 2025SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulationcited by 58
  7. 2026MEM: Multi-Scale Embodied Memory for Vision Language Action Modelscited by 55
  8. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  9. 2024A Dual Process VLA: Efficient Robotic Manipulation Leveraging VLMcited by 44
  10. 2025Running VLAs at Real-time Speedcited by 42
  11. 2026RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policiescited by 39
  12. 2024From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Controlcited by 34
  13. 2025UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representationscited by 30
  14. 2025Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoningcited by 28
  15. 2026LaST0: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Modelcited by 28
  16. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 25
  17. 2026LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queriescited by 24
  18. 2026NitroGen: An Open Foundation Model for Generalist Gaming Agentscited by 22
  19. 2024Language-Conditioned Robotic Manipulation with Fast and Slow Thinkingcited by 17
  20. 2025ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulationcited by 13
  21. 2026RoboTTT: Context Scaling for Robot Policiescited by 13
  22. 2026What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agentscited by 9
  23. 2026ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulationcited by 8
  24. 2026StreamVLA: Breaking the Reason-Act Cycle via Completion-State Gatingcited by 7
  25. 2026Libra-VLA: Achieving Learning Equilibrium via Asynchronous Coarse-to-Fine Dual-Systemcited by 6
  26. 2025See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrationscited by 4
  27. 2026Latent Bridge: Feature Delta Prediction for Efficient Dual-System Vision-Language-Action Model Inferencecited by 4
  28. 2025D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AIcited by 3
  29. 2026SeeTraceAct: Visibility-Aware Latent Planning from Cross-Embodiment Demonstration Videoscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution
  2. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
  3. 2026MemBodied: Recurrent Associative Memory for Vision-Language-Action Models
  4. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model
  5. 2025Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learning
  6. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
  7. 2026RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models
  8. 2023Vision-Language Foundation Models as Effective Robot Imitators