Data to 7 Oct 2026

DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution

arXiv 2411.02359 ↗First posted 4 Nov 2024cs.RO cs.AI cs.LGcited by 116

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more

Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

MLLMs have demonstrated remarkable comprehension and reasoning capabilities with complex language and visual data. These advances have spurred the vision of establishing a generalist robotic MLLM proficient in understanding complex human instructions and accomplishing various embodied tasks. However, developing MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. In contrast, the inference of MLLMs involves storing billions of parameters and performing tremendous computation, imposing significant hardware demands. In our paper, we propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR-VLA, or simply DeeR) that automatically adjusts the size of the activated MLLM based on each situation at hand. The approach leverages a multi-exit architecture in MLLMs, which allows the model to terminate processing once a proper size of the model has been activated for a specific situation, thus avoiding further redundant computation. Additionally, we develop novel algorithms that establish early-termination criteria for DeeR, conditioned on predefined demands such as average computational cost (i.e., power consumption), as well as peak computational consumption (i.e., latency) and GPU memory usage. These enhancements ensure that DeeR operates efficiently under varying resource constraints while maintaining competitive performance. On the CALVIN robot manipulation benchmark, DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance. Code and checkpoints are available at https://github.com/yueyang130/DeeR-VLA.

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 64 of its 77 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

64 of 77 references.

Closest ancestors · 1

Same direction, newest first

2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May

Other references · 63

2023GPT-4 Technical Reportcited by 18,132
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2023Visual Instruction Tuningcited by 7,792
2016Layer Normalizationcited by 5,579
2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
2024Octo: An Open-Source Generalist Robot Policycited by 1,427

38 more

2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
2021Dynamic Neural Networks: A Surveycited by 405
2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
2022Confident Adaptive Language Modelingcited by 297
2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
2019Depth-Adaptive Transformercited by 227
2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
2024A Survey on Efficient Inference for Large Language Modelscited by 197
2023Efficient Large Language Models: A Surveycited by 183
2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
2017Adaptive Neural Networks for Efficient Inferencecited by 161
2020Resolution Adaptive Networks for Efficient Inferencecited by 146
2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 51; 2026: 62.

116 papers.

Sort:

By area:

2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 140
2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 89
2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 88
2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 83
2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 69
2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64

38 more

2025SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Accelerationcited by 56
2025FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policiescited by 56
2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
2025VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulationcited by 48
2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulationcited by 48
2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
2025SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruningcited by 39
2025CEED-VLA: Consistency Vision-Language-Action Model with Early-Exit Decodingcited by 38
2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
2025Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Processcited by 37
2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
2025A Survey on Efficient Vision-Language-Action Modelscited by 34
2025Action-aware Dynamic Pruning for Efficient Vision-Language-Action Manipulationcited by 32
2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptancecited by 31
2025Agentic Robot: A Brain-Inspired Framework for Vision-Language-Action Models in Embodied Agentscited by 30
2025Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoningcited by 28
2025GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulationcited by 26
2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Surveycited by 25
2025VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Votingcited by 22
2025SemanticVLA: Semantic-Aligned Sparsification and Enhancement for Efficient Robotic Manipulationcited by 20
2026FASTER: Rethinking Real-Time Flow VLAscited by 20
2026How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perfcited by 17
2025Bridging the Semantic-Action Gap in Visual Token Pruning for Efficient VLA Inferencecited by 14
2025CombatVLA: An Efficient Vision-Language-Action Model for Combat Tasks in 3D Action Role-Playing Gamescited by 13
2025On-Device Diffusion Transformer Policy for Efficient Robot Manipulationcited by 13
2025EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfercited by 12
2025HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policiescited by 10
2026DyQ-VLA: Temporal-Dynamic-Aware Quantization for Embodied Vision-Language-Action Modelscited by 10
2026MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generationcited by 10
2026Fast-dVLA: Accelerating Discrete Diffusion VLA to Real-Time Performancecited by 9
2026Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflectioncited by 9
2026AT-VLA: Adaptive Tactile Injection for Enhanced Feedback Reaction in Vision-Language-Action Modelscited by 9
2026MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Modelscited by 9
2024ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesiscited by 8
2026A1: A Fully Transparent Open-Source, Adaptive and Efficient Truncated Vision-Language-Action Modelcited by 8
2026Shallow-π: Knowledge Distillation for Flow-based VLAscited by 7
2026CRAFT: Adapting VLA Models to Contact-rich Manipulation via Force-aware Curriculum Fine-tuningcited by 7

50 most cited of 116 All 116 (JSON)

Similar, not cited

Close in wording, no citation either way