WhisperSpeech
Open-source TTS by inverting Whisper: semantic tokens from Whisper's encoder, acoustic modeling from EnCodec, high-fidelity output via Vocos — trained on 80,000+ hours of speech.
About
WhisperSpeech (Collabora, 2023) is the open-source text-to-speech system built by running Whisper backwards. The pipeline is a two-stage token design in the style of AudioLM and SPEAR-TTS: Whisper's encoder block produces embeddings that get quantized into semantic tokens (which means any language Whisper understands needs only raw audio, no transcripts, to bootstrap), Meta's EnCodec models the acoustic tokens, and Charactr's Vocos vocoder renders the final waveform. Collabora scaled it to 80,000+ hours of speech and talked through the lessons in conference presentations. The published samples skew English (with a Polish-voice English demo showing cross-lingual voice transfer), and full multilingual training was still on the roadmap — so treat it as English-first. Pretrained models are on Hugging Face with a pip package, and the project is fully self-hostable. It has been quiet since 2024, but as a no-API, no-cost TTS you can run offline, it remains a solid pick.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- No formal benchmarks published — quality evidence is the project's audio samples and conference presentations\n- No independent benchmark replication found
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 7/10
Scored 7/10 (TIER verified, metrics-v1). Evidence: open code and weights on Hugging Face/GitHub; conference talks documenting 80k-hour training; pip-installable. Strengths: fully open and self-hostable, clever Whisper-inversion design, multilingual bootstrapping without transcripts. Docked for: (1) project quiet since 2024 — maintenance is effectively stalled; (2) no formal benchmarks, only samples; (3) multilingual training never finished, so English-first in practice. License: open source (check the repo for the exact license before commercial use).
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 6.0/10
- Maintenance · recency, releases, issue load
- 4.0/10
- Openness · license + docs
- 9.0/10
- Manual review · sanity read + red-flag check
- 7.0/10
- · Whisper-inverted TTS: Whisper encoder semantic tokens + EnCodec acoustic + Vocos vocoder
- · Trained on 80k+ hours (per conference talks); pip package + Hugging Face weights
- · No formal benchmarks — samples and talks only; no independent replication
- · Upstream quiet since 2024; multilingual training never finished (English-first in practice)