# 3D Object Manipulation in a Single Image using Generative Models

arXiv: https://arxiv.org/abs/2501.12935 · HTML: https://mlsanity.com/paper/2501.12935/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 52 of its 61 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets](https://mlsanity.com/paper/2406.13897/) (2024)
- [UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models](https://mlsanity.com/paper/2302.04867/) (2023)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)
- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion](https://mlsanity.com/paper/2402.03162/) (2024)
- [Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image](https://mlsanity.com/paper/2405.20343/) (2024)
- [Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models](https://mlsanity.com/paper/2312.13913/) (2023)

## Built on by (most cited first, 7 of 7)

- [In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer](https://mlsanity.com/paper/2504.20690/) (2025)
- [RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control](https://mlsanity.com/paper/2502.10059/) (2025)
- [A Physical Coherence Benchmark for Evaluating Video Generation Models via Optical Flow-guided Frame Prediction](https://mlsanity.com/paper/2502.05503/) (2025)
- [Generative Blocks World: Moving Things Around in Pictures](https://mlsanity.com/paper/2506.20703/) (2025)
- [Stroke3D: Lifting 2D strokes into rigged 3D model via latent diffusion models](https://mlsanity.com/paper/2602.09713/) (2026)
- [TRACE: High-Fidelity 3D Scene Editing via Tangible Reconstruction and Geometry-Aligned Contextual Video Masking](https://mlsanity.com/paper/2604.01207/) (2026)
- [Preserving Source Video Realism: High-Fidelity Face Swapping for Cinematic Quality](https://mlsanity.com/paper/2512.07951/) (2025)

## Abstract (arXiv)

Object manipulation in images aims to not only edit the object's presentation but also gift objects with motion. Previous methods encountered challenges in concurrently handling static editing and dynamic generation, while also struggling to achieve fidelity in object appearance and scene lighting. In this work, we introduce OMG3D, a novel framework that integrates the precise geometric control with the generative power of diffusion models, thus achieving significant enhancements in visual performance. Our framework first converts 2D objects into 3D, enabling user-directed modifications and lifelike motions at the geometric level. To address texture realism, we propose CustomRefiner, a texture refinement module that pre-train a customized diffusion model, aligning the details and style of coarse renderings of 3D rough model with the original image, further refine the texture. Additionally, we introduce IllumiCombiner, a lighting processing module that estimates and corrects background lighting to match human visual perception, resulting in more realistic shadow effects. Extensive experiments demonstrate the outstanding visual performance of our approach in both static and dynamic scenarios. Remarkably, all these steps can be done using one NVIDIA 3090. Project page is at https://whalesong-zrs.github.io/OMG3D-projectpage/