# CoMa: Contextual Massing Generation with Vision-Language Models

arXiv: https://arxiv.org/abs/2601.08464 · HTML: https://mlsanity.com/paper/2601.08464/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 56 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [StarVector: Generating Scalable Vector Graphics Code from Images and Text](https://mlsanity.com/paper/2312.11556/) (2023)
- [House-GAN: Relational Generative Adversarial Networks for Graph-constrained House Layout Generation](https://mlsanity.com/paper/2003.06988/) (2020)
- [Why Warmup the Learning Rate? Underlying Mechanisms and Improvements](https://mlsanity.com/paper/2406.09405/) (2024)
- [Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Space](https://mlsanity.com/paper/2503.11094/) (2025)
- [SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition](https://mlsanity.com/paper/2511.21471/) (2025)
- [Large Language Models for Computer-Aided Design: A Survey](https://mlsanity.com/paper/2505.08137/) (2025)
- [Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior](https://mlsanity.com/paper/2404.06780/) (2024)
- [Graph2Plan: Learning Floorplan Generation from Layout Graphs](https://mlsanity.com/paper/2004.13204/) (2020)
- [GlobalMapper: Arbitrary-Shaped Urban Layout Generation](https://mlsanity.com/paper/2307.09693/) (2023)
- [House-GAN++: Generative Adversarial Layout Refinement Networks](https://mlsanity.com/paper/2103.02574/) (2021)
- [Generative AI for Urban Planning: Synthesizing Satellite Imagery via Diffusion Models](https://mlsanity.com/paper/2505.08833/) (2025)
- [Generative Design through Quality-Diversity Data Synthesis and Language Models](https://mlsanity.com/paper/2405.09997/) (2024)
- [Generative AI for Urban Design: A Stepwise Approach Integrating Human Expertise with Multimodal Diffusion Models](https://mlsanity.com/paper/2505.24260/) (2025)

## Abstract (arXiv)

Context-aware building massing is an important early-stage design task: given a site for buildings, a generated massing should not only fit the target parcel, but also relate to the scale, density, and morphology of its surrounding urban fabric. This task is naturally multimodal, since the target output should remain structured and editable, while the surrounding context, including other buildings or roads, can be represented as vector geometry, map imagery, or three-dimensional views. In this paper, we study contextual massing generation using vision-language models (VLMs) and analyze their performance on this task across different context modalities during training and inference. We assemble an experimental dataset of 12,845 Melbourne massings with parcel contours, structured 3D geometry, neighboring buildings, top-down views, and multi-view 3D context images. We also introduce a learned contextual relevance metric for evaluating whether generated massings are morphologically compatible with their surrounding context. Using Qwen3-VL models, we compare no-context, unimodal-context, and multimodal-context training regimes and evaluate inference performance under controlled combinations of modalities and amounts of context. The results show that model size strongly affects generation quality, multimodal training improves the use of individual modalities, and multimodal inference provides a stronger contextual signal than isolated context inputs.