# Model Adaptation for Time Constrained Embodied Control

arXiv: https://arxiv.org/abs/2406.11128 · HTML: https://mlsanity.com/paper/2406.11128/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 44 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 30 of 36)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?](https://mlsanity.com/paper/2303.18240/) (2023)
- [Adaptive Neural Networks for Efficient Inference](https://mlsanity.com/paper/1702.07811/) (2017)
- [Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/1703.06182/) (2017)
- [Multi-Task Reinforcement Learning with Soft Modularization](https://mlsanity.com/paper/2003.13661/) (2020)
- [Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning](https://mlsanity.com/paper/1812.00971/) (2018)
- [Less is More: Task-aware Layer-wise Distillation for Language Model Compression](https://mlsanity.com/paper/2210.01351/) (2022)
- [Dynamic Slimmable Network](https://mlsanity.com/paper/2103.13258/) (2021)
- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [Channel Gating Neural Networks](https://mlsanity.com/paper/1805.12549/) (2018)
- [Multi-Target Embodied Question Answering](https://mlsanity.com/paper/1904.04686/) (2019)
- [Vision-Language Navigation with Random Environmental Mixup](https://mlsanity.com/paper/2106.07876/) (2021)
- [Learning Object Relation Graph and Tentative Policy for Visual Navigation](https://mlsanity.com/paper/2007.11018/) (2020)
- [DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following](https://mlsanity.com/paper/2202.13330/) (2022)
- [Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution](https://mlsanity.com/paper/1701.00299/) (2017)
- [Discrete Sequential Prediction of Continuous Actions for Deep RL](https://mlsanity.com/paper/1705.05035/) (2017)
- [Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inference](https://mlsanity.com/paper/1907.04523/) (2019)
- [SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigation](https://mlsanity.com/paper/1905.07512/) (2019)
- [Knowledge-based Embodied Question Answering](https://mlsanity.com/paper/2109.07872/) (2021)
- [Latency-aware Spatial-wise Dynamic Networks](https://mlsanity.com/paper/2210.06223/) (2022)
- [Interactron: Embodied Adaptive Object Detection](https://mlsanity.com/paper/2202.00660/) (2022)
- [Controllable Dynamic Multi-Task Architectures](https://mlsanity.com/paper/2203.14949/) (2022)

## Abstract (arXiv)

When adopting a deep learning model for embodied agents, it is required that the model structure be optimized for specific tasks and operational conditions. Such optimization can be static such as model compression or dynamic such as adaptive inference. Yet, these techniques have not been fully investigated for embodied control systems subject to time constraints, which necessitate sequential decision-making for multiple tasks, each with distinct inference latency limitations. In this paper, we present MoDeC, a time constraint-aware embodied control framework using the modular model adaptation. We formulate model adaptation to varying operational conditions on resource and time restrictions as dynamic routing on a modular network, incorporating these conditions as part of multi-task objectives. Our evaluation across several vision-based embodied environments demonstrates the robustness of MoDeC, showing that it outperforms other model adaptation methods in both performance and adherence to time constraints in robotic manipulation and autonomous driving applications