# Multimodal Behavior Tree Generation: A Small Vision-Language Model for Robot Task Planning

arXiv: https://arxiv.org/abs/2603.06084 · HTML: https://mlsanity.com/paper/2603.06084/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 23 of 23)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [SmolVLM: Redefining small and efficient multimodal models](https://mlsanity.com/paper/2504.05299/) (2025)
- [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023)
- [BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation](https://mlsanity.com/paper/2403.09227/) (2024)
- [Behavior Trees in Robotics and AI: An Introduction](https://mlsanity.com/paper/1709.00084/) (2017)
- [The Marathon 2: A Navigation System](https://mlsanity.com/paper/2003.00368/) (2020)
- [A Survey of Behavior Trees in Robotics and AI](https://mlsanity.com/paper/2005.05842/) (2020)
- [LLM-BRAIn: AI-driven Fast Generation of Robot Behaviour Tree based on Large Language Model](https://mlsanity.com/paper/2305.19352/) (2023)
- [LLM-as-BT-Planner: Leveraging LLMs for Behavior Tree Generation in Robot Task Planning](https://mlsanity.com/paper/2409.10444/) (2024)
- [BTGenBot: Behavior Tree Generation for Robotic Tasks with Lightweight LLMs](https://mlsanity.com/paper/2403.12761/) (2024)
- [LLM-BT: Performing Robotic Adaptive Tasks based on Large Language Models and Behavior Trees](https://mlsanity.com/paper/2404.05134/) (2024)
- [Behavior Trees and State Machines in Robotics Applications](https://mlsanity.com/paper/2208.04211/) (2022)
- [Integrating Intent Understanding and Optimal Behavior Planning for Behavior Tree Generation from Human Instructions](https://mlsanity.com/paper/2405.07474/) (2024)
- [Automatic Behavior Tree Expansion with LLMs for Robotic Manipulation](https://mlsanity.com/paper/2409.13356/) (2024)
- [An expressiveness hierarchy of Behavior Trees and related architectures](https://mlsanity.com/paper/2104.07919/) (2021)
- [VLM-driven Behavior Tree for Context-aware Task Planning](https://mlsanity.com/paper/2501.03968/) (2025)
- [BTGenBot-2: Efficient Behavior Tree Generation with Small Language Models](https://mlsanity.com/paper/2602.01870/) (2026)
- [Video-to-BT: Generating Reactive Behavior Trees from Human Demonstration Videos for Robotic Assembly](https://mlsanity.com/paper/2509.16611/) (2025)

## Built on by (most cited first, 2 of 2)

- [Learning Structured Robot Policies from Vision-Language Models via Synthetic Neuro-Symbolic Supervision](https://mlsanity.com/paper/2604.02812/) (2026)
- [Physico-Geospatial Grounded Scene Interpretation for Mobile Robotics](https://mlsanity.com/paper/2609.06629/) (2026)

## Abstract (arXiv)

Large language models have been widely used for robotic task planning, often taking advantage of representations such as Behavior Trees (BTs). Vision-Language Models (VLMs) have extended these works by grounding the generated plans in the observed scene. However, existing methods are either text-only or rely on large proprietary VLMs, while no dataset pairs visual observations and task instructions with executable and ROS2-compatible BTs. We address this gap with a multi-stage teacher pipeline that converts 1,622 Open X-Embodiment episodes into an augmented multimodal BT dataset containing 2,433 examples. We use this dataset to fine-tune compact and open-source VLMs, ranging from 500M to 4B parameters, using parameter-efficient fine-tuning (PEFT). We then evaluate the generated BTs offline in terms of syntactic correctness and by executing them on 15 household tasks in BEHAVIOR-1K. Our best model, Gemma-3 4B, achieves perfect BT validity and an 87% success rate, outperforming Claude Opus 4.8 and approaching GPT-5, while running locally. Finally, our ablation studies show that adding visual observations increases task success from 40% to 87%, while data augmentation increases BT validity from 65% to 100%.