mlsanityData to 14 Oct 2026

Enhancing Product Search Interfaces with Sketch-Guided Diffusion and Language Agents

arXiv 2504.08739 ↗First posted 21 Mar 2025cs.IR cs.HCcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

The rapid progress in diffusion models, transformers, and language agents has unlocked new possibilities, yet their potential in user interfaces and commercial applications remains underexplored. We present Sketch-Search Agent, a novel framework that transforms the image search experience by integrating a multimodal language agent with freehand sketches as control signals for diffusion models. Using the T2I-Adapter, Sketch-Search Agent combines sketches and text prompts to generate high-quality query images, encoded via a CLIP image encoder for efficient matching against an image corpus. Unlike existing methods, Sketch-Search Agent requires minimal setup, no additional training, and excels in sketch-based image retrieval and natural language interactions. The multimodal agent enhances user experience by dynamically retaining preferences, ranking results, and refining queries for personalized recommendations. This interactive design empowers users to create sketches and receive tailored product suggestions, showcasing the potential of diffusion models in user-centric image retrieval. Experiments confirm Sketch-Search Agent's high accuracy in delivering relevant product search results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 20 of its 36 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

20 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance11 Jun
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov

Other references · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  4. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  5. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  6. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 266
  7. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 245
  8. 2024TradingAgents: Multi-Agents LLM Financial Trading Frameworkcited by 188
  9. 2021Deep Learning for Instance Retrieval: A Surveycited by 70
  10. 2023CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agentscited by 65
  11. 2024eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Datacited by 46
  12. 2024Cross-conditioned Diffusion Model for Medical Image to Image Translationcited by 26
5 more
  1. 2024ChatShop: Interactive Information Seeking with Language Agentscited by 24
  2. 2024ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understandingcited by 23
  3. 2018Web-Scale Responsive Visual Search at Bingcited by 13
  4. 2024RNA-GPT: Multimodal Generative System for RNA Sequence Understandingcited by 8
  5. 2024Enabling Generative Design Tools with LLM Agents for Mechanical Computation Devices: A Case Studycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024It's All About Your Sketch: Democratising Sketch Control in Diffusion Models
  2. 2024Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers
  3. 2025StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback
  4. 2024SketchAgent: Language-Driven Sequential Sketch Generation
  5. 2024Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping
  6. 2025FUSE : Failure-aware Usage of Subagent Evidence for MultiModal Search and Recommendation
  7. 2024Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines
  8. 2024Leveraging Large Language Models for Multimodal Search