# ORACLE: Leveraging Mutual Information for Consistent Character Generation with LoRAs in Diffusion Models

arXiv: https://arxiv.org/abs/2406.02820 · HTML: https://mlsanity.com/paper/2406.02820/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation](https://mlsanity.com/paper/2405.01434/) (2024)
- [TripoSR: Fast 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/2403.02151/) (2024)
- [RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models](https://mlsanity.com/paper/2312.04524/) (2023)
- [Make-A-Story: Visual Memory Conditioned Consistent Story Generation](https://mlsanity.com/paper/2211.13319/) (2022)
- [TaleCrafter: Interactive Story Visualization with Multiple Characters](https://mlsanity.com/paper/2305.18247/) (2023)
- [The Chosen One: Consistent Characters in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2311.10093/) (2023)
- [Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion Models](https://mlsanity.com/paper/2302.03900/) (2023)

## Built on by (most cited first, 7 of 7)

- [Diffusion Self-Distillation for Zero-Shot Customized Image Generation](https://mlsanity.com/paper/2411.18616/) (2024)
- [Visual Persona: Foundation Model for Full-Body Human Customization](https://mlsanity.com/paper/2503.15406/) (2025)
- [CharCom: Composable Identity Control for Multi-Character Story Illustration](https://mlsanity.com/paper/2510.10135/) (2025)
- [Plot'n Polish: Zero-shot Story Visualization and Disentangled Editing with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2509.04446/) (2025)
- [Consistent text-to-image generation via scene de-contextualization](https://mlsanity.com/paper/2510.14553/) (2025)
- [LouvreSAE: Sparse Autoencoders for Interpretable and Controllable Style Transfer](https://mlsanity.com/paper/2512.18930/) (2025)
- [AttriStory: Fine-grained Attribute Realization for Visual Storytelling with Diffusion Models](https://mlsanity.com/paper/2605.20777/) (2026)

## Abstract (arXiv)

Text-to-image diffusion models have recently taken center stage as pivotal tools in promoting visual creativity across an array of domains such as comic book artistry, children's literature, game development, and web design. These models harness the power of artificial intelligence to convert textual descriptions into vivid images, thereby enabling artists and creators to bring their imaginative concepts to life with unprecedented ease. However, one of the significant hurdles that persist is the challenge of maintaining consistency in character generation across diverse contexts. Variations in textual prompts, even if minor, can yield vastly different visual outputs, posing a considerable problem in projects that require a uniform representation of characters throughout. In this paper, we introduce a novel framework designed to produce consistent character representations from a single text prompt across diverse settings. Through both quantitative and qualitative analyses, we demonstrate that our framework outperforms existing methods in generating characters with consistent visual identities, underscoring its potential to transform creative industries. By addressing the critical challenge of character consistency, we not only enhance the practical utility of these models but also broaden the horizons for artistic and creative expression.