mlsanityData to 14 Oct 2026

Populate-A-Scene: Affordance-Aware Human Video Generation

arXiv 2507.00334 ↗First posted 1 Jul 2025cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Can a video generation model be repurposed as an interactive world simulator? We explore the affordance perception potential of text-to-video models by teaching them to predict human-environment interaction. Given a scene image and a prompt describing human actions, we fine-tune the model to insert a person into the scene, while ensuring coherent behavior, appearance, harmonization, and scene affordance. Unlike prior work, we infer human affordance for video generation (i.e., where to insert a person and how they should behave) from a single scene image, without explicit conditions like bounding boxes or body poses. An in-depth study of cross-attention heatmaps demonstrates that we can uncover the inherent affordance perception of a pre-trained video model without labeled affordance datasets.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 48 of its 67 references that are in the map. First posted on arXiv 1 Jul 2025.

Builds on

48 of 67 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023Segment Anythingcited by 8,234
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2022Flow Matching for Generative Modelingcited by 5,702
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  10. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
36 more
  1. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  2. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  3. 2021Grounded Language-Image Pre-trainingcited by 1,156
  4. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  5. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 688
  6. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  7. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  8. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  9. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  10. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 388
  11. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  12. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  13. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  14. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  15. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 272
  16. 2023Demystifying CLIP Datacited by 270
  17. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  18. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  19. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 154
  20. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  21. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 132
  22. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  23. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 80
  24. 2017Learning to Act Properly: Predicting and Explaining Affordances from Imagescited by 74
  25. 2025Animate Anyone 2: High-Fidelity Character Image Animation with Environment Affordancecited by 61
  26. 2023Putting People in Their Place: Affordance-Aware Human Insertion into Scenescited by 53
  27. 2023One-Shot Open Affordance Learning with Foundation Modelscited by 45
  28. 2018Binge Watching: Scaling Affordance Learning from Sitcomscited by 41
  29. 2023GenZI: Zero-Shot 3D Human-Scene Interaction Generationcited by 40
  30. 2023CoTDet: Affordance Knowledge Prompting for Task Driven Object Detectioncited by 36
  31. 2023Affordance Grounding from Demonstration Video to Target Imagecited by 34
  32. 2021One-Shot Affordance Detectioncited by 32
  33. 2023DAT++: Spatially Dynamic Vision Transformer with Deformable Attentioncited by 21
  34. 2023SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Controlcited by 18
  35. 2024InVi: Object Insertion In Videos Using Off-the-Shelf Diffusion Modelscited by 10
  36. 2023Animating Street Viewcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Video Generation Models: A Survey of Post-Training and Alignmentcited by 0