mlsanityData to 7 Oct 2026

GazeFusion: Saliency-Guided Image Generation

arXiv 2407.04191 ↗First posted 16 Mar 2024cs.CV cs.AI cs.GRcited by 4

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion models offer unprecedented image generation power given just a text prompt. While emerging approaches for controlling diffusion models have enabled users to specify the desired spatial layouts of the generated content, they cannot predict or control where viewers will pay more attention due to the complexity of human vision. Recognizing the significance of attention-controllable image generation in practical applications, we present a saliency-guided framework to incorporate the data priors of human visual attention mechanisms into the generation process. Given a user-specified viewer attention distribution, our control module conditions a diffusion model to generate images that attract viewers' attention toward the desired regions. To assess the efficacy of our approach, we performed an eye-tracked user study and a large-scale model-based saliency analysis. The results evidence that both the cross-user eye gaze distributions and the saliency models' predictions align with the desired attention distributions. Lastly, we outline several applications, including interactive design of saliency guidance, attention suppression in unwanted regions, and adaptive generation for varied display/viewing conditions.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 28 of its 71 references that are in the map. First posted on arXiv 16 Mar 2024.

Builds on

28 of 71 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  2. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  3. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar

Other references · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  7. 2021No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistencycited by 165
  8. 2015Salient Object Detection: A Benchmarkcited by 130
  9. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  10. 2022AUC Maximization in the Era of Big Data and AI: A Surveycited by 100
  11. 2020Unified Image and Video Saliency Modelingcited by 74
  12. 2018Revisiting Video Saliency: A Large-scale Benchmark and a New Modelcited by 66
13 more
  1. 2017SalGAN: Visual Saliency Prediction with Generative Adversarial Networkscited by 63
  2. 2014Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNetcited by 58
  3. 2019TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detectioncited by 58
  4. 2020Uncertainty Inspired RGB-D Saliency Detectioncited by 56
  5. 2021Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusioncited by 56
  6. 2021DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modelingcited by 55
  7. 2015CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Researchcited by 52
  8. 2018EML-NET:An Expandable Multi-Layer NETwork for Saliency Predictioncited by 51
  9. 2021ScanGAN360: A Generative Model of Realistic Scanpaths for 360circ Imagescited by 23
  10. 2021Deep Saliency Prior for Reducing Visual Distractioncited by 17
  11. 2020Look here! A parametric learning based approach to redirect visual attentioncited by 12
  12. 2017BubbleView: an interface for crowdsourcing image importance maps and tracking visual attentioncited by 3
  13. 2023Toward Optimized VR/AR Ergonomics: Modeling and Predicting User Neck Muscle Contractioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Controllable Audio-Visual Viewpoint Generation from 360deg Spatial Informationcited by 0
  2. 2025Exploring Object-Aware Attention Guided Frame Association for RGB-D SLAMcited by 0
  3. 2025EEG-Driven Image Reconstruction with Saliency-Guided Diffusion Modelscited by 0
  4. 2026Amplify What You Gaze At: Target Saliency Boosting in Text-to-Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Saliency Guided Optimization of Diffusion Latents
  2. 2026Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation
  3. 2026Infinite Gaze Generation for Videos with Autoregressive Diffusion
  4. 2024Layout Control and Semantic Guidance with Attention Loss Backward for T2I Diffusion Model
  5. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
  6. 2024Enhancing Image Layout Control with Loss-Guided Diffusion Models
  7. 2025Controllable Coupled Image Generation via Diffusion Models
  8. 2024DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360deg Images