# Enhancing Intent Understanding for Ambiguous prompt: A Human-Machine Co-Adaption Strategy

arXiv: https://arxiv.org/abs/2501.15167 · HTML: https://mlsanity.com/paper/2501.15167/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 80 references that are in the map.
- First posted on arXiv 25 Jan 2025.

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [Aligning Text-to-Image Models using Human Feedback](https://mlsanity.com/paper/2302.12192/) (2023)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods](https://mlsanity.com/paper/2403.20150/) (2024)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting](https://mlsanity.com/paper/2212.06909/) (2022)
- [StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1710.10916/) (2017)
- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [Rich Human Feedback for Text-to-Image Generation](https://mlsanity.com/paper/2312.10240/) (2023)
- [DUET: Dual Clustering Enhanced Multivariate Time Series Forecasting](https://mlsanity.com/paper/2412.10859/) (2024)
- [ChatGPT is not all you need. A State of the Art Review of large Generative AI models](https://mlsanity.com/paper/2301.04655/) (2023)
- [Panacea: Pareto Alignment via Preference Adaptation for LLMs](https://mlsanity.com/paper/2402.02030/) (2024)

## Built on by (most cited first, 16 of 16)

- [FloorPlan-DeepSeek (FPDS): A multimodal approach to floorplan generation using vector-based next room prediction](https://mlsanity.com/paper/2506.21562/) (2025)
- [Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs?](https://mlsanity.com/paper/2503.10838/) (2025)
- [Segment Any Architectural Facades (SAAF):An automatic segmentation model for building facades, walls and windows based on multimodal semantics guidance](https://mlsanity.com/paper/2506.09071/) (2025)
- [ArchiLense: A Framework for Quantitative Analysis of Architectural Styles Based on Vision Large Language Models](https://mlsanity.com/paper/2506.07739/) (2025)
- [MI-PRUN: Optimize Large Language Model Pruning via Mutual Information](https://mlsanity.com/paper/2601.07212/) (2026)
- [EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer](https://mlsanity.com/paper/2605.01732/) (2026)
- [OMR-Diffusion:Optimizing Multi-Round Enhanced Training in Diffusion Models for Improved Intent Understanding](https://mlsanity.com/paper/2503.17660/) (2025)
- [Optimizing Multi-Round Enhanced Training in Diffusion Models for Improved Preference Understanding](https://mlsanity.com/paper/2504.18204/) (2025)
- [UrbanSense:A Framework for Quantitative Analysis of Urban Streetscapes leveraging Vision Large Language Models](https://mlsanity.com/paper/2506.10342/) (2025)
- [ArchShapeNet:An Interpretable 3D-CNN Framework for Evaluating Architectural Shapes](https://mlsanity.com/paper/2506.14832/) (2025)
- [EvoMail: Self-Evolving Cognitive Agents for Adaptive Spam and Phishing Email Defense](https://mlsanity.com/paper/2509.21129/) (2025)
- [A Deformable Attention-Based Detection Transformer with Cross-Scale Feature Fusion for Industrial Coil Spring Inspection](https://mlsanity.com/paper/2603.13429/) (2026)
- [JND-Guided Neural Watermarking with Spatial Transformer Decoding for Screen-Capture Robustness](https://mlsanity.com/paper/2603.26766/) (2026)
- [Though Language Models Err While They Strive: Conformal Prediction for Self-Correcting Scientific Generation](https://mlsanity.com/paper/2607.16704/) (2026)
- [Code Consistency Preference Optimization Verification for Language Model Alignment](https://mlsanity.com/paper/2609.19002/) (2026)
- [CompileRover: Revolutionizing Virtual Machine Compiler Optimization with a Tri-Role LLM-Driven Framework](https://mlsanity.com/paper/2609.19004/) (2026)

## Abstract (arXiv)

Current image generation systems produce high-quality images but struggle with ambiguous user prompts, making interpretation of actual user intentions difficult. Many users must modify their prompts several times to ensure the generated images meet their expectations. While some methods focus on enhancing prompts to make the generated images fit user needs, the model is still hard to understand users' real needs, especially for non-expert users. In this research, we aim to enhance the visual parameter-tuning process, making the model user-friendly for individuals without specialized knowledge and better understand user needs. We propose a human-machine co-adaption strategy using mutual information between the user's prompts and the pictures under modification as the optimizing target to make the system better adapt to user needs. We find that an improved model can reduce the necessity for multiple rounds of adjustments. We also collect multi-round dialogue datasets with prompts and images pairs and user intent. Various experiments demonstrate the effectiveness of the proposed method in our proposed dataset. Our dataset and annotation tools will be available.