mlsanityData to 14 Oct 2026

LCM-Lookahead for Encoder-based Text-to-Image Personalization

arXiv 2404.03620 ↗First posted 4 Apr 2024cs.CV cs.GRcited by 48

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent advancements in diffusion models have introduced fast sampling methods that can effectively produce high-quality images in just one or a few denoising steps. Interestingly, when these are distilled from existing diffusion models, they often maintain alignment with the original model, retaining similar outputs for similar prompts and seeds. These properties present opportunities to leverage fast sampling methods as a shortcut-mechanism, using them to create a preview of denoised outputs through which we can backpropagate image-space losses. In this work, we explore the potential of using such shortcut-mechanisms to guide the personalization of text-to-image models to specific facial identities. We focus on encoder-based personalization approaches, and demonstrate that by tuning them with a lookahead identity loss, we can achieve higher identity fidelity, without sacrificing layout diversity or prompt alignment. We further explore the use of attention sharing mechanisms and consistent data generation for the task of personalization, and find that encoder training can benefit from both.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 53 of its 84 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

53 of 84 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024BootPIG: Bootstrapping Zero-shot Personalized Image Generation Capabilities in Pretrained Diffusion Models25 Jan
  2. 2023PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization11 Dec
  3. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  4. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models5 Jul
  5. 2023Inserting Anybody in Diffusion Models via Celeb Basis1 Jun
  6. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention17 May
  7. 2023Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models5 Apr
  8. 2023ReVersion: Diffusion-Based Relation Inversion from Images23 Mar
  9. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuning20 Mar
  10. 2023DIRE for Diffusion-Generated Image Detection16 Mar
  11. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models23 Feb
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
2 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
27 more
  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  2. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  4. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  5. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  6. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  7. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  8. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 544
  9. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  10. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 459
  11. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 442
  12. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  13. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 328
  14. 2020CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognitioncited by 321
  15. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 225
  16. 2023Style Aligned Image Generation via Shared Attentioncited by 216
  17. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 212
  18. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  19. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  20. 2023FreeU: Free Lunch in Diffusion U-Netcited by 196
  21. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  22. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 136
  23. 2023A Neural Space-Time Representation for Text-to-Image Personalizationcited by 123
  24. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 108
  25. 2023BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrappingcited by 98
  26. 2023Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Modelscited by 85
  27. 2018ArcFace: Additive Angular Margin Loss for Deep Face Recognitioncited by 56

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 26; 2026: 4.

48 papers.

Sort:

By area:

  1. 2024PuLID: Pure and Lightning ID Customization via Contrastive Alignmentcited by 144
  2. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 124
  3. 2025DreamO: A Unified Framework for Image Customizationcited by 83
  4. 2024TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Modelscited by 64
  5. 2024MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generationcited by 51
  6. 2024Diffusion Self-Distillation for Zero-Shot Customized Image Generationcited by 47
  7. 2024One Diffusion to Generate Them Allcited by 41
  8. 2024PersonalVideo: High ID-Fidelity Video Customization without Dynamic and Semantic Degradationcited by 29
  9. 2025Personalize Anything for Free with Diffusion Transformercited by 23
  10. 2025Nested Attention: Semantic-aware Attention Values for Concept Personalizationcited by 21
  11. 2024Robust CLIP-Based Detector for Exposing Diffusion Model-Generated Imagescited by 20
  12. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
36 more
  1. 2025UniVG: A Generalist Diffusion Model for Unified Image Generation and Editingcited by 19
  2. 2024Omni-ID: Holistic Identity Representation Designed for Generative Taskscited by 18
  3. 2025MagicID: Hybrid Preference Optimization for ID-Consistent and Dynamic-Preserved Video Customizationcited by 17
  4. 2025Cut2Next: Generating Next Shot via In-Context Tuningcited by 15
  5. 2024Unbounded: A Generative Infinite Game of Character Life Simulationcited by 14
  6. 2024InstantRestore: Single-Step Personalized Face Restoration with Shared-Image Attentioncited by 13
  7. 2024RectifID: Personalizing Rectified Flow with Anchored Classifier Guidancecited by 11
  8. 2025DynamicID: Zero-Shot Multi-ID Image Personalization with Flexible Facial Editabilitycited by 10
  9. 2025HyperLoRA: Parameter-Efficient Adaptive Generation for Portrait Synthesiscited by 10
  10. 2025Tight Inversion: Image-Conditioned Inversion for Real Image Editingcited by 7
  11. 2025Visual Persona: Foundation Model for Full-Body Human Customizationcited by 7
  12. 2025ComposeMe: Attribute-Specific Image Prompts for Controllable Human Image Generationcited by 7
  13. 2024Controllable Human Image Generation with Personalized Multi-Garmentscited by 5
  14. 2026Spanning the Visual Analogy Space with a Weight Basis of LoRAscited by 5
  15. 2025FaceID-6M: A Large-Scale, Open-Source FaceID Customization Datasetcited by 4
  16. 2024SerialGen: Personalized Image Generation by First Standardization Then Personalizationcited by 3
  17. 2025IP-Prompter: Training-Free Theme-Specific Image Generation via Dynamic Visual Promptingcited by 3
  18. 2025AgeBooth: Controllable Facial Aging and Rejuvenation via Diffusion Modelscited by 3
  19. 2025CanvasComposer: Personalized Group Photo Generation via a Multi-Reference Canvascited by 3
  20. 2025In-Context Sync-LoRA for Portrait Video Editingcited by 3
  21. 2024FPGA: Flexible Portrait Generation Approachcited by 2
  22. 2024Foundation Cures Personalization: Improving Personalized Models' Prompt Consistency via Hidden Foundation Knowledgecited by 2
  23. 2025Stencil: Subject-Driven Generation with Context Guidancecited by 2
  24. 2024RepControlNet: ControlNet Reparameterizationcited by 1
  25. 2024MotionCharacter: Fine-Grained Motion Controllable Human Video Generationcited by 1
  26. 2025Turn That Frown Upside Down: FaceID Customization via Cross-Training Datacited by 1
  27. 2025Meta-LoRA: Meta-Learning LoRA Components for Domain-Aware ID Personalizationcited by 1
  28. 2025ExpertGen: Training-Free Expert Guidance for Controllable Text-to-Face Generationcited by 1
  29. 2025FontAdapter: Instant Font Adaptation in Visual Text Generationcited by 1
  30. 2025Adversarial Concept Distillation for One-Step Diffusion Personalizationcited by 1
  31. 2025Single Image Iterative Subject-driven Generation and Editingcited by 0
  32. 2025MagicView: Multi-View Consistent Identity Customization via Priors-Guided In-Context Learningcited by 0
  33. 2025Training for Identity, Inference for Controllability: A Unified Approach to Tuning-Free Face Personalizationcited by 0
  34. 2026Visual Personalization Turing Testcited by 0
  35. 2026Diff-PC: Identity-preserving and 3D-aware Controllable Diffusion for Zero-shot Portrait Customizationcited by 0
  36. 2026Latent-Identity Tuning in Text-to-Image Personalization Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MasterWeaver: Taming Editability and Face Identity for Personalized Text-to-Image Generation
  2. 2023When StyleGAN Meets Stable Diffusion: a mathscrW_+ Adapter for Personalized Image Generation
  3. 2025Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift
  4. 2025Reverse Personalization
  5. 2024InstantID: Zero-shot Identity-Preserving Generation in Seconds
  6. 2024Infinite-ID: Identity-preserved Personalization via ID-semantics Decoupling Paradigm
  7. 2024Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization
  8. 2024Face2Diffusion for Fast and Editable Face Personalization