# Promptus: Can Prompts Streaming Replace Video Streaming with Stable Diffusion

arXiv: https://arxiv.org/abs/2405.20032 · HTML: https://mlsanity.com/paper/2405.20032/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 61 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [Real-Time Intermediate Flow Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2011.06294/) (2020)
- [Any-to-Any Generation via Composable Diffusion](https://mlsanity.com/paper/2305.11846/) (2023)
- [Scaling Up Vision-Language Pre-training for Image Captioning](https://mlsanity.com/paper/2111.12233/) (2021)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [Neural Video Compression with Feature Modulation](https://mlsanity.com/paper/2402.17414/) (2024)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Generative Image Dynamics](https://mlsanity.com/paper/2309.07906/) (2023)
- [Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement](https://mlsanity.com/paper/2003.01966/) (2020)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity](https://mlsanity.com/paper/2305.11675/) (2023)
- [StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation](https://mlsanity.com/paper/2312.12491/) (2023)
- [Wireless Semantic Communications for Video Conferencing](https://mlsanity.com/paper/2204.07790/) (2022)
- [Extreme Generative Image Compression by Learning Text Embedding from Diffusion Models](https://mlsanity.com/paper/2211.07793/) (2022)
- [Blind Video Deflickering by Neural Filtering with a Flawed Atlas](https://mlsanity.com/paper/2303.08120/) (2023)

## Built on by (most cited first, 3 of 3)

- [CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference](https://mlsanity.com/paper/2604.06036/) (2026)
- [Morphe: High-Fidelity Generative Video Streaming with Vision Foundation Model](https://mlsanity.com/paper/2602.03529/) (2026)
- [PromptMobile: Efficient Promptus for Low Bandwidth Mobile Video Streaming](https://mlsanity.com/paper/2503.16112/) (2025)

## Abstract (arXiv)

With the exponential growth of video traffic, traditional video streaming systems are approaching their limits in compression efficiency and communication capacity. To further reduce bitrate while maintaining quality, we propose Promptus, a disruptive semantic communication system that streaming prompts instead of video content, which represents real-world video frames with a series of "prompts" for delivery and employs Stable Diffusion to generate videos at the receiver. To ensure that the generated video is pixel-aligned with the original video, a gradient descent-based prompt fitting framework is proposed. Further, a low-rank decomposition-based bitrate control algorithm is introduced to achieve adaptive bitrate. For inter-frame compression, an interpolation-aware fitting algorithm is proposed. Evaluations across various video genres demonstrate that, compared to H.265, Promptus can achieve more than a 4x bandwidth reduction while preserving the same perceptual quality. On the other hand, at extremely low bitrates, Promptus can enhance the perceptual quality by 0.139 and 0.118 (in LPIPS) compared to VAE and H.265, respectively, and decreases the ratio of severely distorted frames by 89.3% and 91.7%. Our work opens up a new paradigm for efficient video communication. Promptus is open-sourced at: https://github.com/JiangkaiWu/Promptus.