# ClevrSkills: Compositional Language and Visual Reasoning in Robotics

arXiv: https://arxiv.org/abs/2411.09052 · HTML: https://mlsanity.com/paper/2411.09052/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 16 of its 50 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 16 of 16)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks](https://mlsanity.com/paper/2108.03272/) (2021)
- [RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot](https://mlsanity.com/paper/2307.00595/) (2023)
- [FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning](https://mlsanity.com/paper/2401.08553/) (2024)
- [Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent](https://mlsanity.com/paper/2402.09844/) (2024)
- [ManiPose: A Comprehensive Benchmark for Pose-aware Object Manipulation in Robotics](https://mlsanity.com/paper/2403.13365/) (2024)

## Built on by (most cited first, 8 of 8)

- [Can Vision-Language Models Answer Face to Face Questions in the Real-World?](https://mlsanity.com/paper/2503.19356/) (2025)
- [Focusing on What Matters: Object-Agent-centric Tokenization for Vision Language Action models](https://mlsanity.com/paper/2509.23655/) (2025)
- [Compose by Focus: Scene Graph-based Atomic Skills](https://mlsanity.com/paper/2509.16053/) (2025)
- [Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models](https://mlsanity.com/paper/2506.06242/) (2025)
- [Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks](https://mlsanity.com/paper/2602.21013/) (2026)
- [From Code to Action: Hierarchical Learning of Diffusion-VLM Policies](https://mlsanity.com/paper/2509.24917/) (2025)
- [Regression Test Selection for Updated Capability Modules in Compositional ML Systems via Atomic-Quality Probes](https://mlsanity.com/paper/2604.26689/) (2026)
- [EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents](https://mlsanity.com/paper/2610.13021/) (2026)

## Abstract (arXiv)

Robotics tasks are highly compositional by nature. For example, to perform a high-level task like cleaning the table a robot must employ low-level capabilities of moving the effectors to the objects on the table, pick them up and then move them off the table one-by-one, while re-evaluating the consequently dynamic scenario in the process. Given that large vision language models (VLMs) have shown progress on many tasks that require high level, human-like reasoning, we ask the question: if the models are taught the requisite low-level capabilities, can they compose them in novel ways to achieve interesting high-level tasks like cleaning the table without having to be explicitly taught so? To this end, we present ClevrSkills - a benchmark suite for compositional reasoning in robotics. ClevrSkills is an environment suite developed on top of the ManiSkill2 simulator and an accompanying dataset. The dataset contains trajectories generated on a range of robotics tasks with language and visual annotations as well as multi-modal prompts as task specification. The suite includes a curriculum of tasks with three levels of compositional understanding, starting with simple tasks requiring basic motor skills. We benchmark multiple different VLM baselines on ClevrSkills and show that even after being pre-trained on large numbers of tasks, these models fail on compositional reasoning in robotics tasks.