# Grasp-Anything: Large-scale Grasp Dataset from Foundation Models

arXiv: https://arxiv.org/abs/2309.09818 · HTML: https://mlsanity.com/paper/2309.09818/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 71 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Builds on (most cited first, 30 of 52)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains](https://mlsanity.com/paper/2212.08333/) (2022)
- [Scaling Robot Learning with Semantically Imagined Experience](https://mlsanity.com/paper/2302.11550/) (2023)
- [A Review of Generalized Zero-Shot Learning Methods](https://mlsanity.com/paper/2011.08641/) (2020)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [ACRONYM: A Large-Scale Grasp Dataset Based on Simulation](https://mlsanity.com/paper/2011.09584/) (2020)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations](https://mlsanity.com/paper/2104.01542/) (2021)

## Built on by (most cited first, 30 of 33)

- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [Language-driven Grasp Detection](https://mlsanity.com/paper/2406.09489/) (2024)
- [Reasoning Grasping via Multimodal Large Language Model](https://mlsanity.com/paper/2402.06798/) (2024)
- [Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy](https://mlsanity.com/paper/2512.11218/) (2025)
- [Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance](https://mlsanity.com/paper/2407.13842/) (2024)
- [RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping](https://mlsanity.com/paper/2507.23734/) (2025)
- [Language-driven Scene Synthesis using Multi-conditional Diffusion Model](https://mlsanity.com/paper/2310.15948/) (2023)
- [Lightweight Language-driven Grasp Detection using Conditional Consistency Model](https://mlsanity.com/paper/2407.17967/) (2024)
- [Generative AI Agents in Autonomous Machines: A Safety Perspective](https://mlsanity.com/paper/2410.15489/) (2024)
- [Spatial RoboGrasp: Generalized Robotic Grasping Control Policy](https://mlsanity.com/paper/2505.20814/) (2025)
- [Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applications](https://mlsanity.com/paper/2409.17727/) (2024)
- [Speeding up 6-DoF Grasp Sampling with Quality-Diversity](https://mlsanity.com/paper/2403.06173/) (2024)
- [ManiBox: Enhancing Embodied Spatial Generalization via Scalable Simulation Data Generations](https://mlsanity.com/paper/2411.01850/) (2024)
- [RoboGrasp: A Universal Grasping Policy for Robust Robotic Control](https://mlsanity.com/paper/2502.03072/) (2025)
- [GraspMamba: A Mamba-based Language-driven Grasp Detection Framework with Hierarchical Feature Learning](https://mlsanity.com/paper/2409.14403/) (2024)
- [Efficient Alignment of Unconditioned Action Prior for Language-conditioned Pick and Place in Clutter](https://mlsanity.com/paper/2503.09423/) (2025)
- [Humanoid Agent via Embodied Chain-of-Action Reasoning with Multimodal Foundation Models for Zero-Shot Loco-Manipulation](https://mlsanity.com/paper/2504.09532/) (2025)
- [Language-driven Grasp Detection with Mask-guided Attention](https://mlsanity.com/paper/2407.19877/) (2024)
- [QDGset: A Large Scale Grasping Dataset Generated with Quality-Diversity](https://mlsanity.com/paper/2410.02319/) (2024)
- [VCoT-Grasp: Grasp Foundation Models with Visual Chain-of-Thought Reasoning for Language-driven Grasp Generation](https://mlsanity.com/paper/2510.05827/) (2025)
- [Language-Guided Grasp Detection with Coarse-to-Fine Learning for Robotic Manipulation](https://mlsanity.com/paper/2512.21065/) (2025)
- [RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation](https://mlsanity.com/paper/2603.14880/) (2026)
- [3D Feature Distillation with Object-Centric Priors](https://mlsanity.com/paper/2406.18742/) (2024)
- [SpikePingpong: Spike Vision-based Fast-Slow Pingpong Robot System](https://mlsanity.com/paper/2506.06690/) (2025)
- [Communication-Efficient Module-Wise Federated Learning for Grasp Pose Detection in Cluttered Environments](https://mlsanity.com/paper/2507.05861/) (2025)
- [A Multi-Level Similarity Approach for Single-View Object Grasping: Matching, Planning, and Fine-Tuning](https://mlsanity.com/paper/2507.11938/) (2025)
- [XGrasp: Gripper-Aware Grasp Detection with Multi-Gripper Data Generation](https://mlsanity.com/paper/2510.11036/) (2025)
- [VLAD-Grasp: Zero-shot Grasp Detection via Vision-Language Models](https://mlsanity.com/paper/2511.05791/) (2025)
- [Affordance-Based Manipulation Planning with Text Goals and Sim-to-Real Generalisation via Real-to-Sim Image Conversion](https://mlsanity.com/paper/2607.11004/) (2026)

## Abstract (arXiv)

Foundation models such as ChatGPT have made significant strides in robotic tasks due to their universal representation of real-world domains. In this paper, we leverage foundation models to tackle grasp detection, a persistent challenge in robotics with broad industrial applications. Despite numerous grasp datasets, their object diversity remains limited compared to real-world figures. Fortunately, foundation models possess an extensive repository of real-world knowledge, including objects we encounter in our daily lives. As a consequence, a promising solution to the limited representation in previous grasp datasets is to harness the universal knowledge embedded in these foundation models. We present Grasp-Anything, a new large-scale grasp dataset synthesized from foundation models to implement this solution. Grasp-Anything excels in diversity and magnitude, boasting 1M samples with text descriptions and more than 3M objects, surpassing prior datasets. Empirically, we show that Grasp-Anything successfully facilitates zero-shot grasp detection on vision-based tasks and real-world robotic experiments. Our dataset and code are available at https://grasp-anything-2023.github.io.