LatentSync

LatentSync, from ByteDance, is an open-source lip-synchronization framework that applies audio-conditioned latent diffusion to rewrite mouth movements in video. Its Temporal REPresentation Alignment (TREPA) system keeps lip motion temporally coherent, producing 720p output — a step up in resolution from earlier GAN-based lip-sync tools.

Code, checkpoints, data-processing pipeline, and training code are all public (GitHub bytedance/LatentSync, Hugging Face ByteDance/LatentSync-1.6), with LatentSync 1.6 needing ~18GB VRAM and 1.5 running in ~8GB. Limitations: clips are short (around 5 seconds per pass for best quality), and diffusion-based sync is slower than real-time GAN approaches — this is a quality-first offline tool, not a streaming one.