mlsanityData to 14 Oct 2026

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models

arXiv 2502.02492 ↗First posted 4 Feb 2025cs.CVcited by 113

Not on the map. Why?

Cited by 54 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite tremendous recent progress, generative video models still struggle to capture real-world motion, dynamics, and physics. We show that this limitation arises from the conventional pixel reconstruction objective, which biases models toward appearance fidelity at the expense of motion coherence. To address this, we introduce VideoJAM, a novel framework that instills an effective motion prior to video generators, by encouraging the model to learn a joint appearance-motion representation. VideoJAM is composed of two complementary units. During training, we extend the objective to predict both the generated pixels and their corresponding motion from a single learned representation. During inference, we introduce Inner-Guidance, a mechanism that steers the generation toward coherent motion by leveraging the model's own evolving motion prediction as a dynamic guidance signal. Notably, our framework can be applied to any video model with minimal adaptations, requiring no modifications to the training data or scaling of the model. VideoJAM achieves state-of-the-art performance in motion coherence, surpassing highly competitive proprietary models while also enhancing the perceived visual quality of the generations. These findings emphasize that appearance and motion can be complementary and, when effectively integrated, enhance both the visual quality and the coherence of video generation. Project website: https://hila-chefer.github.io/videojam-paper.github.io/

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). Builds on 42 of its 119 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

42 of 119 references.

References · 42

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2022Classifier-Free Diffusion Guidancecited by 5,847
  6. 2022Flow Matching for Generative Modelingcited by 5,702
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  10. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
30 more
  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  5. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  6. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  7. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  8. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  9. 2023ModelScope Text-to-Video Technical Reportcited by 658
  10. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  11. 2024LTX-Video: Realtime Video Latent Diffusioncited by 581
  12. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  13. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  14. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  15. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  16. 2022UL2: Unifying Language Learning Paradigmscited by 295
  17. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  18. 2023Demystifying CLIP Datacited by 270
  19. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  20. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  21. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 167
  22. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generationcited by 137
  23. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 100
  24. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generationcited by 91
  25. 2023HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusioncited by 68
  26. 2024Still-Moving: Customized Video Generation without Customized Video Datacited by 50
  27. 2024MotionCraft: Physics-based Zero-Shot Video Generationcited by 48
  28. 2023The Hidden Language of Diffusion Modelscited by 34
  29. 2024MotiF: Making Text Count in Image Animation with Motion Focal Losscited by 10
  30. 2023Decouple Content and Motion for Conditional Image-to-Video Generationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 57; 2026: 56.

113 papers.

Sort:

By area:

  1. 2025Motus: A Unified Latent Action World Modelcited by 251
  2. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 192
  3. 2025Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compressioncited by 85
  4. 2025FlowVLA: Visual Chain of Thought-based Motion Reasoning for Vision-Language-Action Modelscited by 50
  5. 2025World-in-World: World Models in a Closed-Loop Worldcited by 47
  6. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 45
  7. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 41
  8. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 40
  9. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 37
  10. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 35
  11. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 34
  12. 2025MoCha: Towards Movie-Grade Talking Character Synthesiscited by 31
38 more
  1. 2025Emergent Temporal Correspondences from Video Diffusion Transformerscited by 31
  2. 2025OmniSync: Towards Universal Lip Synchronization via Diffusion Transformerscited by 27
  3. 2025OmniVDiff: Omni Controllable Video Diffusion for Generation and Understandingcited by 22
  4. 2025FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generationcited by 22
  5. 2025GeoVideo: Introducing Geometric Regularization into Video Generation Modelcited by 19
  6. 2025TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibilitycited by 18
  7. 2025WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perceptioncited by 16
  8. 2025Synthetic Video Enhances Physical Fidelity in Video Synthesiscited by 15
  9. 2025LuxDiT: Lighting Estimation with Video Diffusion Transformercited by 15
  10. 2026Self-Refining Video Samplingcited by 15
  11. 2025EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformercited by 13
  12. 2025SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarioscited by 12
  13. 2026Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulationcited by 12
  14. 2026Phantom: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamicscited by 12
  15. 2025DisMo: Disentangled Motion Representations for Open-World Motion Transfercited by 10
  16. 2025SMRABooth: Subject and Motion Representation Alignment for Customized Video Generationcited by 10
  17. 2026Olaf-World: Orienting Latent Actions for Video World Modelingcited by 10
  18. 2025Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initializationcited by 9
  19. 2026VOID: Video Object and Interaction Deletioncited by 9
  20. 2026What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusioncited by 9
  21. 2025Towards Physical Understanding in Video Generation: A 3D Point Regularization Approachcited by 8
  22. 2025MoGAN: Improving Motion Quality in Video Diffusion via Few-Step Motion Adversarial Post-Trainingcited by 8
  23. 2025WorldPack: Dynamic Frame Compression for Long-context Video World Modelingcited by 8
  24. 2026DreamWAM: Beyond RGB Future Prediction for World Action Modelscited by 8
  25. 2025Target-Aware Video Diffusion Modelscited by 7
  26. 2025UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generationcited by 7
  27. 2026Future Optical Flow Prediction Improves Robot Control & Video Generationcited by 7
  28. 2026DreamWorld: Unified World Modeling in Video Generationcited by 7
  29. 2026VideoWeave: Unlocking Geometric Consistency in Video Generation via Joint Geometry-Video Modelingcited by 7
  30. 2025CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusioncited by 6
  31. 2026CoMoVi: Co-Generation of 3D Human Motions and Realistic Videoscited by 6
  32. 2026Reward-Forcing: Autoregressive Video Generation with Reward Feedbackcited by 6
  33. 2025MusicInfuser: Making Video Diffusion Listen and Dancecited by 5
  34. 2025Point Prompting: Counterfactual Tracking with Video Diffusion Modelscited by 5
  35. 2025SeeU: Seeing the Unseen World via 4D Dynamics-aware Generationcited by 5
  36. 2025WorldReel: 4D Video Generation with Consistent Geometry and Motion Modelingcited by 5
  37. 2025REGLUE Your Latents with Global and Local Semantics for Entangled Diffusioncited by 5
  38. 2026Forecasting Motion in the Wildcited by 5

50 most cited of 113 All 113 (JSON)