# Tell2Design: A Dataset for Language-Guided Floor Plan Generation

arXiv: https://arxiv.org/abs/2311.15941 · HTML: https://mlsanity.com/paper/2311.15941/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 61 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [CM3: A Causal Masked Multimodal Model of the Internet](https://mlsanity.com/paper/2201.07520/) (2022)
- [Generating Images from Captions with Attention](https://mlsanity.com/paper/1511.02793/) (2015)
- [Object-driven Text-to-Image Synthesis via Adversarial Training](https://mlsanity.com/paper/1902.10740/) (2019)
- [HairCLIP: Design Your Hair by Text and Reference Image](https://mlsanity.com/paper/2112.05142/) (2021)

## Built on by (most cited first, 19 of 19)

- [Large Language Models for Planning: A Comprehensive and Systematic Survey](https://mlsanity.com/paper/2505.19683/) (2025)
- [FlairGPT: Repurposing LLMs for Interior Designs](https://mlsanity.com/paper/2501.04648/) (2025)
- [Generating 3D House Wireframes with Semantics](https://mlsanity.com/paper/2407.12267/) (2024)
- [MANSION: Multi-floor lANguage-to-3D Scene generatIOn for loNg-horizon tasks](https://mlsanity.com/paper/2603.11554/) (2026)
- [Constrained Layout Generation with Factor Graphs](https://mlsanity.com/paper/2404.00385/) (2024)
- [HouseTune: Two-Stage Floorplan Generation with LLM Assistance](https://mlsanity.com/paper/2411.12279/) (2024)
- [FloorPlan-DeepSeek (FPDS): A multimodal approach to floorplan generation using vector-based next room prediction](https://mlsanity.com/paper/2506.21562/) (2025)
- [MCP4IFC: IFC-Based Building Design Using Large Language Models](https://mlsanity.com/paper/2511.05533/) (2025)
- [Co-Layout: LLM-driven Co-optimization for Interior Layout](https://mlsanity.com/paper/2511.12474/) (2025)
- [World Craft: Agentic Framework to Create Visualizable Worlds via Text](https://mlsanity.com/paper/2601.09150/) (2026)
- [Tokenization Allows Multimodal Large Language Models to Understand, Generate and Edit Architectural Floor Plans](https://mlsanity.com/paper/2603.11640/) (2026)
- [PlanCraft: Sketch, Refine, and Furnish for Architect-Inspired Progressive 3D Residential Scene Generation](https://mlsanity.com/paper/2607.23491/) (2026)
- [UrbanSense:A Framework for Quantitative Analysis of Urban Streetscapes leveraging Vision Large Language Models](https://mlsanity.com/paper/2506.10342/) (2025)
- [IndoorWorld: Integrating Physical Task Solving and Social Simulation in A Heterogeneous Multi-Agent Environment](https://mlsanity.com/paper/2506.12331/) (2025)
- [GreenPlanner: Practical Floorplan Layout Generation via an Energy-Aware and Function-Feasible Generative Framework](https://mlsanity.com/paper/2512.00406/) (2025)
- [Unified Vector Floorplan Generation via Markup Representation](https://mlsanity.com/paper/2604.04859/) (2026)
- [Generative Floor Plan Design with LLMs via Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2605.14117/) (2026)
- [HypergraphFormer: Learning Hypergraphs from LLMs for Editable Floor Plan Generation](https://mlsanity.com/paper/2605.18932/) (2026)
- [GRE-Diff: Gaussian Room Embeddings for Structured Layout Diffusion](https://mlsanity.com/paper/2607.08086/) (2026)

## Abstract (arXiv)

We consider the task of generating designs directly from natural language descriptions, and consider floor plan generation as the initial research area. Language conditional generative models have recently been very successful in generating high-quality artistic images. However, designs must satisfy different constraints that are not present in generating artistic images, particularly spatial and relational constraints. We make multiple contributions to initiate research on this task. First, we introduce a novel dataset, textitTell2Design (T2D), which contains more than 80k floor plan designs associated with natural language instructions. Second, we propose a Sequence-to-Sequence model that can serve as a strong baseline for future research. Third, we benchmark this task with several text-conditional image generation models. We conclude by conducting human evaluations on the generated samples and providing an analysis of human performance. We hope our contributions will propel the research on language-guided design generation forward.