StreamSpeech
An all-in-one streaming translator: one model that does offline and simultaneous ASR, speech translation, and speech synthesis — and shows you its intermediate transcripts while it works.
About
StreamSpeech (ACL 2024, arXiv 2406.03049) tackles the double problem of simultaneous speech-to-speech translation: translating is hard enough, but a streaming model also needs a policy for when to speak versus when to keep listening. Its answer is multi-task learning in a single unified model — one network jointly learns translation and the simultaneous policy, and the same checkpoint can run offline or streaming ASR, speech-to-text translation, speech-to-speech translation, and TTS. The authors report state-of-the-art results on the CVSS benchmark for both offline and simultaneous S2ST, and a genuinely useful touch: during live translation it surfaces intermediate ASR and text-translation results, so you can watch what it heard and how it's rendering it instead of staring at silence. Code and model weights are published (code under MIT per project mirrors), which makes it one of the few simultaneous-S2ST systems you can actually run and probe yourself.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- State-of-the-art on CVSS for offline and simultaneous S2ST (author-reported, ACL 2024)\n- No independent benchmark replication found
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 7/10
Scored 7/10 (TIER verified, metrics-v1). Evidence: ACL 2024 peer-reviewed paper (arXiv 2406.03049) with SOTA CVSS claims; published code (MIT) and weights you can run. Strengths: rare open simultaneous-S2ST system, all-in-one offline+streaming ASR/ST/TTS, live intermediate results. Docked for: (1) SOTA claim is author-reported with no independent replication; (2) academic project — adoption and maintenance are modest compared with industrial releases. License: code MIT (per project mirrors); check the repo for weight terms.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 6.0/10
- Maintenance · recency, releases, issue load
- 5.0/10
- Openness · license + docs
- 9.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · ACL 2024 (arXiv 2406.03049); multi-task simultaneous S2ST with joint translation+policy learning
- · SOTA on CVSS is author-reported; no independent replication
- · Rare open simultaneous-S2ST system: published code (MIT) and weights, all-in-one offline+streaming ASR/ST/TTS
- · Check repo for weight license terms