Mini-Omni2
Open-source GPT-4o-style omni model: vision, speech and text with duplex interruption on a compact backbone.
About
Mini-Omni2 is the gpt-omni team's open attempt at a GPT-4o-style assistant that sees, hears and talks inside one model. It keeps Qwen2 0.5B as the backbone, adds a CLIP visual encoder and the Whisper encoder for audio, and generates text and speech side by side with a delayed-parallel trick so it can start talking while it is still thinking. A command-based interruption mechanism lets the user cut in mid-sentence, which is what makes the conversation feel full-duplex rather than turn-based.
The team deliberately trained on minimal new data to show that modality expansion does not need exponential dataset growth, and they released the VoiceAssistant-400K dataset alongside the model. That makes it one of the more practical open recipes for streaming voice chat, and the Gradio and Streamlit demos run locally. The catch is scale: a 0.5B backbone caps reasoning quality, published evaluations are author-reported only, and most serious voice products have moved to closed APIs. Best used as a research base or a hackable local voice assistant, not as a frontier model.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- Author-reported evaluations in the paper (arXiv 2410.11190); no independent third-party eval found
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 6/10
Scored 6/10 (TIER metrics, metrics-v1). Evidence: arXiv 2410.11190 (Oct 2024) from the gpt-omni team; GitHub gpt-omni/mini-omni is the established repo for the line (Mini-Omni2 released 2024-10 per its README); VoiceAssistant-400K dataset on HF. Strengths: one of the earliest open end-to-end duplex speech models, open code/data/recipe, real local demos. Docked for: (1) 0.5B backbone limits reasoning quality; (2) evaluations are author-reported only; (3) HF weights distribution for the v2 checkpoint is unclear (HF API lookup inconclusive Oct 2026). License: Apache 2.0 (GitHub repo).
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 4.0/10
- Maintenance · recency, releases, issue load
- 4.0/10
- Openness · license + docs
- 7.0/10
- Manual review · sanity read + red-flag check
- 6.0/10
- · arXiv 2410.11190 (v1 2024-10-15) from the gpt-omni team; Qwen2 0.5B + CLIP + Whisper encoder
- · Mini-Omni2 released 2024-10 per the gpt-omni/mini-omni README; VoiceAssistant-400K dataset released on HF
- · HF API lookup for gpt-omni/mini-omni2 inconclusive (Oct 2026); weights distribution path unclear
- · Evaluations author-reported only; no independent third-party eval found
- · 0.5B backbone caps reasoning quality vs newer closed voice APIs