Freeze-Omni
Low-latency speech-to-speech dialogue on a fully frozen text LLM: duplex voice chat without catastrophic forgetting.
About
Freeze-Omni is Tencent Youtu Lab's take on speech-to-speech dialogue, and its core idea is restraint: keep the text LLM completely frozen and bolt a streaming speech encoder and an autoregressive speech decoder around it. The paper (arXiv 2411.00774, November 2024) reports that the whole system was trained on plain ASR and TTS text-speech pairs plus only 60,000 multi-turn text Q&A examples on 8 GPUs, yet spoken answers stay at roughly the same intelligence level as the backbone LLM's text output. A chunk-level state predictor watches for user interruptions, which is what gives it genuine full-duplex turn-taking instead of push-to-talk.
The frozen-backbone trick is the real contribution: it sidesteps the catastrophic forgetting that shows up when you fine-tune a small LLM on limited speech data, and it means almost any text LLM could be given ears and a voice the same way. The tradeoff is maturity. Code and weights are public (VITA-MLLM on GitHub and Hugging Face), but the GitHub license file restricts use to research only, there are no published benchmark tables to point at, and real-world adoption is thin so far. Treat it as a research platform for duplex voice interfaces, not a production voice stack.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
No published benchmarks found
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 6/10
Scored 6/10 (TIER metrics, metrics-v1). Evidence: arXiv 2411.00774 (Nov 2024) from Tencent Youtu Lab; HF repo VITA-MLLM/Freeze-Omni (22 likes, HF API Oct 2026); project page freeze-omni.github.io live. Strengths: novel frozen-LLM approach to duplex speech, open code and weights, cheap training recipe (60k Q&A on 8 GPUs). Docked for: (1) no published benchmark numbers, all quality claims are qualitative in the paper; (2) license ambiguity: HF repo tags Apache 2.0 but the GitHub License.txt restricts use to research only per third-party index; (3) near-zero adoption. License: Apache 2.0 (HF tags); research-only terms in GitHub License.txt.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 3.0/10
- Maintenance · recency, releases, issue load
- 5.0/10
- Openness · license + docs
- 7.0/10
- Manual review · sanity read + red-flag check
- 6.0/10
- · arXiv 2411.00774 (v1 2024-11-01) from Tencent Youtu Lab; project page freeze-omni.github.io live (curl 200, Oct 2026)
- · HF repo VITA-MLLM/Freeze-Omni: 22 likes, ~0 downloads (HF API Oct 2026)
- · Frozen-LLM training recipe: ASR/TTS text-speech pairs + 60k multi-turn text Q&A on 8 GPUs (paper)
- · No published benchmark numbers; quality claims qualitative in the paper
- · Near-zero adoption; license ambiguity (HF Apache 2.0 tag vs research-only GitHub License.txt)