# Bayesian Optimization for Controlled Image Editing via LLMs

arXiv: https://arxiv.org/abs/2502.18116 · HTML: https://mlsanity.com/paper/2502.18116/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [Pre-trained Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2109.08215/) (2021)
- [ZONE: Zero-Shot Instruction-Guided Local Editing](https://mlsanity.com/paper/2312.16794/) (2023)
- [CAR: Controllable Autoregressive Modeling for Visual Generation](https://mlsanity.com/paper/2410.04671/) (2024)
- [CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs](https://mlsanity.com/paper/2404.01343/) (2024)
- [Intrinsic Entropy of Context Length Scaling in LLMs](https://mlsanity.com/paper/2502.01481/) (2025)
- [Graph Canvas for Controllable 3D Scene Generation](https://mlsanity.com/paper/2412.00091/) (2024)
- [Latent Space Bayesian Optimization with Latent Data Augmentation for Enhanced Exploration](https://mlsanity.com/paper/2302.02399/) (2023)
- [Crowd Counting in Harsh Weather using Image Denoising with Pix2Pix GANs](https://mlsanity.com/paper/2310.07245/) (2023)
- [Maximizing Uncertainty for Federated learning via Bayesian Optimisation-based Model Poisoning](https://mlsanity.com/paper/2501.08002/) (2025)

## Built on by (most cited first, 11 of 11)

- [CountLLM: Towards Generalizable Repetitive Action Counting via Large Language Model](https://mlsanity.com/paper/2503.17690/) (2025)
- [RAM: Recover Any 3D Human Motion in-the-Wild](https://mlsanity.com/paper/2603.19929/) (2026)
- [Attention Consistency for LLMs Explanation](https://mlsanity.com/paper/2509.17178/) (2025)
- [RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders](https://mlsanity.com/paper/2508.05289/) (2025)
- [PSGS: Text-driven Panorama Sliding Scene Generation via Gaussian Splatting](https://mlsanity.com/paper/2602.00463/) (2026)
- [Coarse-to-Fine Personalized LLM Impressions for Streamlined Radiology Reports](https://mlsanity.com/paper/2508.15845/) (2025)
- [CNSocialDepress: A Chinese Social Media Dataset for Depression Risk Detection and Structured Analysis](https://mlsanity.com/paper/2510.11233/) (2025)
- [BCL: Bayesian In-Context Learning Framework for Information Extraction](https://mlsanity.com/paper/2606.18620/) (2026)
- [Adaptive Prompt Elicitation for Text-to-Image Generation](https://mlsanity.com/paper/2602.04713/) (2026)
- [E2CAR: An Efficient 2D-CNN Framework for Real-Time EEG Artifact Removal on Edge Devices](https://mlsanity.com/paper/2602.09035/) (2026)
- [Grounding Multi-Hop Reasoning in Structural Causal Models via Group Relative Policy Optimization](https://mlsanity.com/paper/2605.01482/) (2026)

## Abstract (arXiv)

In the rapidly evolving field of image generation, achieving precise control over generated content and maintaining semantic consistency remain significant limitations, particularly concerning grounding techniques and the necessity for model fine-tuning. To address these challenges, we propose BayesGenie, an off-the-shelf approach that integrates Large Language Models (LLMs) with Bayesian Optimization to facilitate precise and user-friendly image editing. Our method enables users to modify images through natural language descriptions without manual area marking, while preserving the original image's semantic integrity. Unlike existing techniques that require extensive pre-training or fine-tuning, our approach demonstrates remarkable adaptability across various LLMs through its model-agnostic design. BayesGenie employs an adapted Bayesian optimization strategy to automatically refine the inference process parameters, achieving high-precision image editing with minimal user intervention. Through extensive experiments across diverse scenarios, we demonstrate that our framework significantly outperforms existing methods in both editing accuracy and semantic preservation, as validated using different LLMs including Claude3 and GPT-4.