LLaMA-Omni
End-to-end speech-language model on Llama-3.1-8B that answers speech with text and speech at once, at ~226ms latency.
About
LLaMA-Omni is an end-to-end speech interaction model built on Llama-3.1-8B-Instruct. Instead of chaining speech recognition, a language model and text-to-speech the way a conventional voice assistant does, it takes spoken instructions directly and produces text and speech responses at the same time. A Whisper-large-v3 encoder handles the input audio and a unit-based HiFi-GAN vocoder renders the output voice, with the authors reporting response latency as low as 226 milliseconds.
The project is notable for how cheap it was to build: the team reports training took under three days on four GPUs. It was published as an ICLR 2025 paper and later followed by LLaMA-Omni2, which moved to Qwen2.5 backbones with a streaming autoregressive speech decoder and was accepted at ACL 2025. The code is Apache-2.0, but anyone using the weights should know they inherit the Llama 3.1 Community License terms from the base model.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- Paper (arXiv 2409.06666): 226ms response latency; reports high-quality speech interaction vs cascaded baselines
- Published as an ICLR 2025 conference paper
- Author-reported; no independent benchmark verification
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 8/10
Scored 8/10 (TIER verified, metrics-v1). Evidence: ICLR 2025 paper (arXiv 2409.06666) with latency and quality claims; Apache-2.0 official repo (ictnlp/LLaMA-Omni, ~3.1k stars); weights on Hugging Face (ICTNLP/Llama-3.1-8B-Omni); ACL 2025 follow-up (LLaMA-Omni2). Strengths: peer-reviewed, strong community uptake, cheap to reproduce. Docked for: (1) weights inherit Llama 3.1 Community License restrictions from the base model; (2) headline claims are author-reported. License: Apache-2.0 (code); Llama 3.1 Community License (weights).
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 8.0/10
- Maintenance · recency, releases, issue load
- 6.0/10
- Openness · license + docs
- 7.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · ICLR 2025 paper (arXiv 2409.06666): 226ms latency, speech in / text+speech out, <3 days on 4 GPUs
- · Official repo ictnlp/LLaMA-Omni, Apache-2.0, ~3.1k stars; weights on HF (ICTNLP/Llama-3.1-8B-Omni)
- · ACL 2025 follow-up LLaMA-Omni2 on Qwen2.5 with streaming AR speech decoder