mlsanityData to 14 Oct 2026

Underspecification in Scene Description-to-Depiction Tasks

arXiv 2210.05815 ↗First posted 11 Oct 2022cs.CV cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Questions regarding implicitness, ambiguity and underspecification are crucial for understanding the task validity and ethical concerns of multimodal image+text systems, yet have received little attention to date. This position paper maps out a conceptual framework to address this gap, focusing on systems which generate images depicting scenes from scene descriptions. In doing so, we account for how texts and images convey meaning differently. We outline a set of core challenges concerning textual and visual ambiguity, as well as risks that may be amplified by ambiguous and underspecified elements. We propose and discuss strategies for addressing these challenges, including generating visually ambiguous images, and generating a set of diverse images.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 39 of its 91 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

39 of 91 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
27 more
  1. 2018Model Cards for Model Reportingcited by 1,080
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  3. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  4. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  5. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  6. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  7. 2018nocaps: novel object captioning at scalecited by 444
  8. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  9. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  10. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  11. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  12. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  13. 2020Experience Grounds Languagecited by 242
  14. 2015Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labelscited by 64
  15. 2020Targeting the Benchmark: On Methodology in Current Natural Language Processing Researchcited by 45
  16. 2016The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narrativescited by 42
  17. 2019Exposing and Correcting the Gender Bias in Image Captioning Datasets and Modelscited by 36
  18. 2020Diversity and Inclusion Metrics in Subset Selectioncited by 27
  19. 2016Stereotyping and Bias in the Flickr30K Datasetcited by 25
  20. 2020Clue: Cross-modal Coherence Modeling for Caption Generationcited by 24
  21. 2020Pragmatic Issue-Sensitive Image Captioningcited by 15
  22. 2019Informative Image Captioning with External Sources of Informationcited by 14
  23. 2017Cross-linguistic differences and similarities in image descriptionscited by 12
  24. 2019CITE: A Corpus of Image-Text Discourse Relationscited by 12
  25. 2022Justice in Misinformation Detection Systems: An Analysis of Algorithms, Stakeholders, and Potential Harmscited by 10
  26. 2016Pragmatic factors in image description: the case of negationscited by 5
  27. 2020On the use of human reference data for evaluating automatic image descriptionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 15; 2024: 8; 2025: 4; 2026: 2.

30 papers.

Sort:

By area:

  1. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  2. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  3. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  4. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 93
  5. 2023The Bias Amplification Paradox in Text-to-Image Generationcited by 70
  6. 2023ITI-GEN: Inclusive Text-to-Image Generationcited by 64
  7. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Imagescited by 55
  8. 2023AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asiacited by 54
  9. 2022DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Modelscited by 51
  10. 2023Inspecting the Geographical Representativeness of Images from Text-to-Image Modelscited by 44
  11. 2024An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevancecited by 36
  12. 2025Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?cited by 34
18 more
  1. 2023I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphorscited by 31
  2. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  3. 2023A Friendly Face: Do Text-to-Image Systems Rely on Stereotypes when the Input is Under-Specified?cited by 25
  4. 2024Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertaintycited by 25
  5. 2023Prompt Expansion for Adaptive Text-to-Image Generationcited by 24
  6. 2023CoBIT: A Contrastive Bi-directional Image-Text Generation Modelcited by 11
  7. 2023Multilingual Conceptual Coverage in Text-to-Image Modelscited by 9
  8. 2023Dealing with Semantic Underspecification in Multimodal NLPcited by 9
  9. 2023Manipulating Embeddings of Stable Diffusion Promptscited by 9
  10. 2023Semantic and Expressive Variation in Image Captions Across Languagescited by 8
  11. 2024Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST!cited by 7
  12. 2024GRADE: Quantifying Sample Diversity in Text-to-Image Modelscited by 5
  13. 2025Fair Foundation Models for Medical Image Analysis: Challenges and Perspectivescited by 3
  14. 2024Modeling the Sacred: Considerations when Using Religious Texts in Natural Language Processingcited by 1
  15. 2025Att-Adapter: A Robust and Precise Domain-Specific Multi-Attributes T2I Diffusion Adapter via Conditional Variational Autoencodercited by 0
  16. 2025The Face of Persuasion: Analyzing Bias and Generating Culture-Aware Adscited by 0
  17. 2026Where did the ambiguity go? Examining how multimodal models interpret polysemous wordscited by 0
  18. 2026Distinguishing Revision and Delayed Elaboration in Incremental Narrative Interpretationcited by 0