# QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning

arXiv: https://arxiv.org/abs/2412.15576 · HTML: https://mlsanity.com/paper/2412.15576/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- Vision-language-action models / [Efficient VLA inference and adaptation](https://mlsanity.com/vla/efficient-vla-inference-and-adaptation/)

## Builds on (most cited first, 20 of 20)

- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation](https://mlsanity.com/paper/2312.13139/) (2023)
- [Vision-Language Foundation Models as Effective Robot Imitators](https://mlsanity.com/paper/2311.01378/) (2023)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [3D Diffuser Actor: Policy Diffusion with 3D Scene Representations](https://mlsanity.com/paper/2402.10885/) (2024)
- [RT-H: Action Hierarchies Using Language](https://mlsanity.com/paper/2403.01823/) (2024)
- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference](https://mlsanity.com/paper/2403.14520/) (2024)
- [QUAR-VLA: Vision-Language-Action Model for Quadruped Robots](https://mlsanity.com/paper/2312.14457/) (2023)
- [Linearizing Large Language Models](https://mlsanity.com/paper/2405.06640/) (2024)
- [GenLoco: Generalized Locomotion Controllers for Quadrupedal Robots](https://mlsanity.com/paper/2209.05309/) (2022)
- [Barkour: Benchmarking Animal-level Agility with Quadruped Robots](https://mlsanity.com/paper/2305.14654/) (2023)
- [SayTap: Language to Quadrupedal Locomotion](https://mlsanity.com/paper/2306.07580/) (2023)
- [Grounding Multimodal Large Language Models in Actions](https://mlsanity.com/paper/2406.07904/) (2024)
- [Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models](https://mlsanity.com/paper/2404.05291/) (2024)
- [Learning Visual Quadrupedal Loco-Manipulation from Demonstrations](https://mlsanity.com/paper/2403.20328/) (2024)
- [LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model](https://mlsanity.com/paper/2404.01331/) (2024)
- [Learning Multiple Gaits within Latent Space for Quadruped Robots](https://mlsanity.com/paper/2308.03014/) (2023)
- [QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds](https://mlsanity.com/paper/2406.16578/) (2024)

## Built on by (most cited first, 9 of 9)

- [VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model](https://mlsanity.com/paper/2509.09372/) (2025)
- [PointVLA: Injecting the 3D World into Vision-Language-Action Models](https://mlsanity.com/paper/2503.07511/) (2025)
- [ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver](https://mlsanity.com/paper/2508.10333/) (2025)
- [VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulation](https://mlsanity.com/paper/2502.13508/) (2025)
- [Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulation](https://mlsanity.com/paper/2508.19958/) (2025)
- [Humanoid-VLA: Towards Universal Humanoid Control with Visual Integration](https://mlsanity.com/paper/2502.14795/) (2025)
- [ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation](https://mlsanity.com/paper/2601.08325/) (2026)
- [RetoVLA: Reusing Register Tokens for Spatial Reasoning in Vision-Language-Action Models](https://mlsanity.com/paper/2509.21243/) (2025)
- [RECAST: Recasting Vision-Language Semantics into an Actionable Cost Map for Robot Navigation](https://mlsanity.com/paper/2609.32595/) (2026)

## Abstract (arXiv)

This paper addresses the inherent inference latency challenges associated with deploying multimodal large language models (MLLM) in quadruped vision-language-action (QUAR-VLA) tasks. Our investigation reveals that conventional parameter reduction techniques ultimately impair the performance of the language foundation model during the action instruction tuning phase, making them unsuitable for this purpose. We introduce a novel latency-free quadruped MLLM model, dubbed QUART-Online, designed to enhance inference efficiency without degrading the performance of the language foundation model. By incorporating Action Chunk Discretization (ACD), we compress the original action representation space, mapping continuous action values onto a smaller set of discrete representative vectors while preserving critical information. Subsequently, we fine-tune the MLLM to integrate vision, language, and compressed actions into a unified semantic space. Experimental results demonstrate that QUART-Online operates in tandem with the existing MLLM system, achieving real-time inference in sync with the underlying controller frequency, significantly boosting the success rate across various tasks by 65%. Our project page is https://quart-online.github.io.