Upscale-A-Video
Text-guided diffusion upscaling for real-world video: SD priors with temporal layers so the hallucinated detail stays consistent across frames.
About
Upscale-A-Video (CVPR 2024 Highlight, arXiv 2312.06640) is the diffusion-based video super-resolution model from Shangchen Zhou's team at NTU S-Lab. It takes the Stable Diffusion x4 upscaler as its backbone and makes it video-safe two ways. Locally, temporal layers are added to the U-Net and the VAE decoder is retrained with spatial-feature-transform conditioning to kill the flicker that VAE decoding introduces. Globally, a training-free flow-guided recurrent latent propagation module fuses latents across the whole sequence so long videos stay coherent.
Because it is diffusion-based, text prompts can steer texture creation and a noise-level control trades fidelity against generation — you decide how much hallucination you want. The authors report it surpassing prior methods on synthetic, real-world, and AI-generated video benchmarks (their numbers), and they released the YouHQ dataset alongside. Inference code arrived in September 2024 with checkpoints via Google Drive and optional LLaVA captioning. It is heavy and slow compared to CNN upscalers, but nothing open matched its generative detail at release.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
Author-reported (CVPR 2024): surpasses prior methods on synthetic and real-world benchmarks and on AI-generated videos
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: CVPR 2024 Highlight paper arXiv 2312.06640 (NTU S-Lab); repo sczhou/Upscale-A-Video confirms inference code release (Sept 2024) and checkpoints. Strengths: strong venue, genuinely novel local+global temporal design, open inference code, text-guided control. Docked for: (1) benchmark claims are author-reported with no independent audit; (2) checkpoint license and redistribution terms are not stated on the repo; (3) diffusion inference is slow and GPU-hungry with no hosted option. License: research code; checkpoint license terms unverified.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 7.0/10
- Maintenance · recency, releases, issue load
- 6.0/10
- Openness · license + docs
- 7.0/10
- Manual review · sanity read + red-flag check
- 7.0/10
- · CVPR 2024 Highlight, arXiv 2312.06640 (NTU S-Lab)
- · Text-guided latent diffusion on SD x4 upscaler; temporal layers in U-Net + VAE decoder; flow-guided latent propagation
- · Inference code released Sept 2024; checkpoints via Google Drive; optional LLaVA captioning
- · Author-reported SOTA; diffusion inference slow; no hosted option