mlsanityData to 7 Oct 2026

Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialogue

arXiv 2106.15550 ↗First posted 29 Jun 2021cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Building an interactive artificial intelligence that can ask questions about the real world is one of the biggest challenges for vision and language problems. In particular, goal-oriented visual dialogue, where the aim of the agent is to seek information by asking questions during a turn-taking dialogue, has been gaining scholarly attention recently. While several existing models based on the GuessWhat?! dataset have been proposed, the Questioner typically asks simple category-based questions or absolute spatial questions. This might be problematic for complex scenes where the objects share attributes or in cases where descriptive questions are required to distinguish objects. In this paper, we propose a novel Questioner architecture, called Unified Questioner Transformer (UniQer), for descriptive question generation with referring expressions. In addition, we build a goal-oriented visual dialogue task called CLEVR Ask. It synthesizes complex scenes that require the Questioner to generate descriptive questions. We train our model with two variants of CLEVR Ask datasets. The results of the quantitative and qualitative evaluations show that UniQer outperforms the baseline.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 52 of its 71 references that are in the map. First posted on arXiv 29 Jun 2021.

Builds on

52 of 71 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2014Conditional Generative Adversarial Netscited by 2,152
  11. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  12. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
38 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  2. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  3. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  4. 2016Modeling Context in Referring Expressionscited by 1,054
  5. 2016Generative Adversarial Text to Image Synthesiscited by 735
  6. 2018Which Training Methods for GANs do actually Converge?cited by 672
  7. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  8. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  9. 2016Visual Dialogcited by 377
  10. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  11. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  12. 2018cGANs with Projection Discriminatorcited by 253
  13. 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 234
  14. 2017Geometric GANcited by 232
  15. 2017Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesiscited by 215
  16. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  17. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  18. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  19. 2020A U-Net Based Discriminator for Generative Adversarial Networkscited by 128
  20. 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 112
  21. 2014From Captions to Visual Concepts and Backcited by 109
  22. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  23. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  24. 2016A Joint Speaker-Listener-Reinforcer Model for Referring Expressionscited by 74
  25. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  26. 2018Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instructioncited by 61
  27. 2017Comprehension-guided referring expressionscited by 52
  28. 2019Factor Graph Attentioncited by 41
  29. 2020SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoningcited by 32
  30. 2020Text as Neural Operator: Image Manipulation by Text Instructioncited by 26
  31. 2016Automatic Generation of Grounded Visual Questionscited by 19
  32. 2017End-to-end optimization of goal-driven and visually grounded dialogue systemscited by 17
  33. 2018Beyond task success: A closer look at jointly learning to see, ask, and GuessWhatcited by 15
  34. 2019Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneckcited by 15
  35. 2019What Should I Ask? Using Conversationally Informative Rewards for Goal-Oriented Visual Dialogcited by 14
  36. 2019Visual Dialogue State Tracking for Question Generationcited by 13
  37. 2017Asking the Difficult Questions: Goal-Oriented Visual Question Generation via Intermediate Rewardscited by 7
  38. 2018A Multimodal Classifier Generative Adversarial Network for Carry and Place Tasks from Ambiguous Language Instructionscited by 7

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 2; 2025: 1.

8 papers.

  1. 2023Object-Centric Learning with Slot Mixture Modulecited by 11
  2. 2025Are We Done with Object-Centric Learning?cited by 10
  3. 2023Alexpaca: Learning Factual Clarification Question Generation Without Examplescited by 2
  4. 2023InViG: Benchmarking Interactive Visual Grounding with 500K Human-Robot Interactionscited by 2
  5. 2024SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interactioncited by 2
  6. 2024Symbolic Disentangled Representations for Imagescited by 2
  7. 2023Selectively Providing Reliance Calibration Cues With Reliance Predictioncited by 1
  8. 2023Pointing out Human Answer Mistakes in a Goal-Oriented Visual Dialoguecited by 1