Budget-Constrained Embodied Perception: Four Resource Walls and a Pre-Registered Evaluation of Access-Structured Perception on Open Models at less than 31B
Not on the map. Why?
Abstract (arXiv)
Embodied multimodal agents must answer from growing observation streams under a fixed per-decision token budget. We formalize this constraint through four resource walls: a perceptual Shannon wall for bounded state, a horizon wall for query-independent frame selection, a round wall for non-adaptive retrieval, and a conditional composition wall for fixed-depth inference. We introduce ASP, a training-free wrapper for frozen multimodal models that combines a capped structured state, a verbatim episodic index, and query-conditioned budget allocation with iterative access. Following a pre-registered protocol, we evaluate seven open-weight models from 3B to 31B on SEW-Bench, a license-free synthetic long-horizon walkthrough benchmark constructed to instantiate these walls. The registered natural-video benchmarks were not run because their frames require dataset agreements; our evidence therefore concerns access mechanisms, not natural-scene perception. Under a 4,096-token decision budget, ASP reaches 75 to 94% episodic retrieval accuracy, compared with 3 to 19% for equal-budget query-independent sampling, and budget reallocation outperforms quadrupling the sampling budget on every backbone. However, the full three-component architecture does not validate channel duality: removing the compressive state raises the flagship mean from 35.4 to 58.0, ASP does not outperform the verbatim-only baseline on any backbone, and two of four pre-registered falsification criteria fire. These results show that query-conditioned access, rather than parameter count or context growth alone, is decisive under a fixed budget, while prompted online compression does not earn its cost in this setting.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 70 of its 80 references that are in the map. First posted on arXiv 24 Aug 2026.
Builds on
70 of 80 references.
References · 70
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
38 more
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2025Gemma 3 Technical Reportcited by 1,686
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
- 2025s1: Simple test-time scalingcited by 1,311
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
- 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
- 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
- 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
- 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
- 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
- 2024Long Context Transfer from Language to Visioncited by 502
- 2025Qwen3-Omni Technical Reportcited by 451
- 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
- 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
- 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
- 2024The Platonic Representation Hypothesiscited by 362
- 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
- 2017Embodied Question Answeringcited by 335
- 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
- 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
- 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
- 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
- 2022Transformers Learn Shortcuts to Automatacited by 260
- 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
- 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
- 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
- 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
50 most cited of 70 All 70 (JSON)
Built on by
Not cited yet.