VoiceCraft

VoiceCraft is a neural codec language model for zero-shot speech editing and text-to-speech, developed by Puyuan Peng and colleagues at UT Austin with Meta FAIR, published on arXiv in March 2024. It frames speech editing as token infilling: a Transformer decoder with a causal-masking plus delayed-stacking rearrangement procedure generates codec tokens conditioned on bidirectional context.

On speech editing, human evaluators rated VoiceCraft's edited speech nearly indistinguishable from unedited recordings for naturalness; on zero-shot TTS it outperformed prior models including VALL-E and XTTS v2. The authors evaluated on realistic in-the-wild data spanning audiobooks, internet videos, and podcasts, and introduced the REALEDIT benchmark for editing evaluation.

Model weights (330M and 830M variants) are openly available, and the project includes Gradio demos. The authors prohibit non-consensual voice generation and editing.