# TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation

arXiv: https://arxiv.org/abs/2512.01204 · HTML: https://mlsanity.com/paper/2512.01204/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 60 references that are in the map.
- First posted on arXiv 1 Dec 2025.

## Builds on (most cited first, 30 of 55)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation](https://mlsanity.com/paper/2506.18088/) (2025)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [3D-FUTURE: 3D Furniture shape with TextURE](https://mlsanity.com/paper/2009.09633/) (2020)
- [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://mlsanity.com/paper/2312.09067/) (2023)
- [Seedream 4.0: Toward Next-generation Multimodal Image Generation](https://mlsanity.com/paper/2509.20427/) (2025)
- [LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes](https://mlsanity.com/paper/2311.13384/) (2023)
- [WonderWorld: Interactive 3D Scene Generation from a Single Image](https://mlsanity.com/paper/2406.09394/) (2024)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Details](https://mlsanity.com/paper/2506.16504/) (2025)
- [MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation](https://mlsanity.com/paper/2412.03558/) (2024)
- [CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image](https://mlsanity.com/paper/2502.12894/) (2025)
- [GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs](https://mlsanity.com/paper/2312.00093/) (2023)
- [LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models](https://mlsanity.com/paper/2412.02193/) (2024)

## Built on by (most cited first, 3 of 3)

- [IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning](https://mlsanity.com/paper/2607.03614/) (2026)
- [CoDimRecon: Agentic Reconstruction of Sim-Ready 3D Scenes with Deformable Curves, Surfaces, and Volumes](https://mlsanity.com/paper/2609.36024/) (2026)
- [ArticuTable: Generating Instance-Level Interactive Rigid-Articulated 3D Tabletop Scenes from a Single Image](https://mlsanity.com/paper/2610.05249/) (2026)

## Abstract (arXiv)

Simulation provides a low-cost, scalable pathway to large-scale robotic manipulation data collection. However, existing 3D scene generation methods can rarely be applied directly to manipulation data synthesis, as their generated scenes often lack instance-level interactivity and physical plausibility. Focusing on tabletop manipulation, we propose TabletopGen, a training-free and automated tabletop scene generation and interactive simulation engine. Starting from text or a single image, we first obtain independent 3D object models via generative instance extraction. Second, we introduce a novel pose and scale alignment approach that recovers a collision-free scene layout using a Differentiable Rotation Optimizer and a Top-View Spatial Alignment mechanism. Finally, we assemble the generated scene in a physics simulator with collision geometry, yielding a stable, interactable environment for synthesizing multimodal manipulation data. Extensive experiments and user studies demonstrate that TabletopGen achieves state-of-the-art performance in visual fidelity, layout accuracy, and physical plausibility. Furthermore, we validate the executability of the collected trajectories on a real robotic arm via zero-shot real-to-sim-to-real policy transfer, indicating that TabletopGen can serve as a reliable data engine for robotic manipulation data synthesis.