mlsanityData to 7 Oct 2026

Budget-Constrained Embodied Perception: Four Resource Walls and a Pre-Registered Evaluation of Access-Structured Perception on Open Models at less than 31B

arXiv 2608.22975 ↗First posted 24 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Embodied multimodal agents must answer from growing observation streams under a fixed per-decision token budget. We formalize this constraint through four resource walls: a perceptual Shannon wall for bounded state, a horizon wall for query-independent frame selection, a round wall for non-adaptive retrieval, and a conditional composition wall for fixed-depth inference. We introduce ASP, a training-free wrapper for frozen multimodal models that combines a capped structured state, a verbatim episodic index, and query-conditioned budget allocation with iterative access. Following a pre-registered protocol, we evaluate seven open-weight models from 3B to 31B on SEW-Bench, a license-free synthetic long-horizon walkthrough benchmark constructed to instantiate these walls. The registered natural-video benchmarks were not run because their frames require dataset agreements; our evidence therefore concerns access mechanisms, not natural-scene perception. Under a 4,096-token decision budget, ASP reaches 75 to 94% episodic retrieval accuracy, compared with 3 to 19% for equal-budget query-independent sampling, and budget reallocation outperforms quadrupling the sampling budget on every backbone. However, the full three-component architecture does not validate channel duality: removing the compressive state raises the flagship mean from 35.4 to 58.0, ASP does not outperform the verbatim-only baseline on any backbone, and two of four pre-registered falsification criteria fire. These results show that query-conditioned access, rather than parameter count or context growth alone, is decisive under a fixed budget, while prompted online compression does not earn its cost in this setting.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 70 of its 80 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

70 of 80 references.

References · 70

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  5. 2025Gemma 3 Technical Reportcited by 1,686
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  8. 2025s1: Simple test-time scalingcited by 1,311
  9. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  12. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  13. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  14. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  15. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  16. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  17. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  18. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  19. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  20. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  21. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  22. 2024Long Context Transfer from Language to Visioncited by 502
  23. 2025Qwen3-Omni Technical Reportcited by 451
  24. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  25. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  26. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  27. 2024The Platonic Representation Hypothesiscited by 362
  28. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  29. 2017Embodied Question Answeringcited by 335
  30. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  31. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  32. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  33. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  34. 2022Transformers Learn Shortcuts to Automatacited by 260
  35. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  36. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  37. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  38. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196

50 most cited of 70 All 70 (JSON)

Built on by

Not cited yet.