# Underspecification in Scene Description-to-Depiction Tasks

arXiv: https://arxiv.org/abs/2210.05815 · HTML: https://mlsanity.com/paper/2210.05815/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 91 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels](https://mlsanity.com/paper/1512.06974/) (2015)
- [Targeting the Benchmark: On Methodology in Current Natural Language Processing Research](https://mlsanity.com/paper/2007.04792/) (2020)
- [The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives](https://mlsanity.com/paper/1611.05118/) (2016)
- [Exposing and Correcting the Gender Bias in Image Captioning Datasets and Models](https://mlsanity.com/paper/1912.00578/) (2019)
- [Diversity and Inclusion Metrics in Subset Selection](https://mlsanity.com/paper/2002.03256/) (2020)

## Built on by (most cited first, 30 of 30)

- [PaLI-X: On Scaling up a Multilingual Vision and Language Model](https://mlsanity.com/paper/2305.18565/) (2023)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [DOCCI: Descriptions of Connected and Contrasting Images](https://mlsanity.com/paper/2404.19753/) (2024)
- [Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task](https://mlsanity.com/paper/2310.09336/) (2023)
- [The Bias Amplification Paradox in Text-to-Image Generation](https://mlsanity.com/paper/2308.00755/) (2023)
- [ITI-GEN: Inclusive Text-to-Image Generation](https://mlsanity.com/paper/2309.05569/) (2023)
- [Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images](https://mlsanity.com/paper/2402.05779/) (2024)
- [AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia](https://mlsanity.com/paper/2305.11844/) (2023)
- [DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Models](https://mlsanity.com/paper/2210.10606/) (2022)
- [Inspecting the Geographical Representativeness of Images from Text-to-Image Models](https://mlsanity.com/paper/2305.11080/) (2023)
- [An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance](https://mlsanity.com/paper/2404.01247/) (2024)
- [Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?](https://mlsanity.com/paper/2509.03516/) (2025)
- [I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors](https://mlsanity.com/paper/2305.14724/) (2023)
- [Beyond Aesthetics: Cultural Competence in Text-to-Image Models](https://mlsanity.com/paper/2407.06863/) (2024)
- [A Friendly Face: Do Text-to-Image Systems Rely on Stereotypes when the Input is Under-Specified?](https://mlsanity.com/paper/2302.07159/) (2023)
- [Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty](https://mlsanity.com/paper/2412.06771/) (2024)
- [Prompt Expansion for Adaptive Text-to-Image Generation](https://mlsanity.com/paper/2312.16720/) (2023)
- [CoBIT: A Contrastive Bi-directional Image-Text Generation Model](https://mlsanity.com/paper/2303.13455/) (2023)
- [Multilingual Conceptual Coverage in Text-to-Image Models](https://mlsanity.com/paper/2306.01735/) (2023)
- [Dealing with Semantic Underspecification in Multimodal NLP](https://mlsanity.com/paper/2306.05240/) (2023)
- [Manipulating Embeddings of Stable Diffusion Prompts](https://mlsanity.com/paper/2308.12059/) (2023)
- [Semantic and Expressive Variation in Image Captions Across Languages](https://mlsanity.com/paper/2310.14356/) (2023)
- [Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST!](https://mlsanity.com/paper/2402.12486/) (2024)
- [GRADE: Quantifying Sample Diversity in Text-to-Image Models](https://mlsanity.com/paper/2410.22592/) (2024)
- [Fair Foundation Models for Medical Image Analysis: Challenges and Perspectives](https://mlsanity.com/paper/2502.16841/) (2025)
- [Modeling the Sacred: Considerations when Using Religious Texts in Natural Language Processing](https://mlsanity.com/paper/2404.14740/) (2024)
- [Att-Adapter: A Robust and Precise Domain-Specific Multi-Attributes T2I Diffusion Adapter via Conditional Variational Autoencoder](https://mlsanity.com/paper/2503.11937/) (2025)
- [The Face of Persuasion: Analyzing Bias and Generating Culture-Aware Ads](https://mlsanity.com/paper/2510.15240/) (2025)
- [Where did the ambiguity go? Examining how multimodal models interpret polysemous words](https://mlsanity.com/paper/2608.00410/) (2026)
- [Distinguishing Revision and Delayed Elaboration in Incremental Narrative Interpretation](https://mlsanity.com/paper/2608.21364/) (2026)

## Abstract (arXiv)

Questions regarding implicitness, ambiguity and underspecification are crucial for understanding the task validity and ethical concerns of multimodal image+text systems, yet have received little attention to date. This position paper maps out a conceptual framework to address this gap, focusing on systems which generate images depicting scenes from scene descriptions. In doing so, we account for how texts and images convey meaning differently. We outline a set of core challenges concerning textual and visual ambiguity, as well as risks that may be amplified by ambiguous and underspecified elements. We propose and discuss strategies for addressing these challenges, including generating visually ambiguous images, and generating a set of diverse images.