TokenSpeed
An open-source LLM inference engine built for agentic workloads, with a C++ control plane and Python execution plane.
About
TokenSpeed is an LLM inference engine from the non-profit LightSeek Foundation, designed specifically for production agentic workloads. Its core architectural bet is splitting the control plane — a C++ finite-state machine where the type system enforces request lifecycle and KV-cache safety at compile time — from the execution plane in Python, which keeps iteration fast for researchers.
Kernels are a first-class modular subsystem with a portable API and centralized registry, including one of the fastest MLA implementations on Blackwell, and the server is vLLM-compatible on parameters. The PyTorch Foundation has highlighted it in its ecosystem landscape, and NVIDIA and AMD both provide hardware and engineering support; the headline throughput numbers (e.g. 685 tokens/s per user on Qwen3.8-Flash-Next) are vendor-reported and workload-specific.
What it can do
- C++ control plane with compile-time KV-cache safety guarantees
- Pluggable kernel registry with fast MLA for Blackwell
- vLLM-level usability with compatible server parameters
- Model recipes and parallelism configuration for serving
- OpenAI-compatible serving endpoint for agent frameworks
Setup
- Follow the getting-started guide at lightseek.org/tokenspeed/guides/getting-started
- Launch a server per the launching guide, picking model recipes for your GPU
- Configure server parameters (vLLM-compatible where documented) and parallelism
- Point your agent framework at the OpenAI-compatible endpoint
Prerequisites
NVIDIA (or AMD) GPUs with recent drivers; familiarity with serving configuration; models in a supported format. Day-0 support has covered Qwen3.8, Kimi K3, and GLM 5.3 Flash releases.
Use cases
Works with
Trust score — 5/10
Scored 7/10 (METRICS tier, metrics-v1). Evidence: MIT license, non-profit foundation backing (neutral vs commercial engines), active development (pushed Oct 17 2026), highlighted by the PyTorch Foundation's ecosystem landscape, with real hardware support from NVIDIA and AMD. Docked for: (1) young project — announced May 2026 — so production hardening history is thin; (2) throughput headlines are vendor-reported on vendor-chosen models and GPUs. License: MIT.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 6.0/10
- Maintenance · recency, releases, issue load
- 7.0/10
- Openness · license + docs
- 9.0/10
- Manual review · sanity read + red-flag check
- 7.0/10
- · 2,672 GitHub stars; 352 forks; GitHub daily trending
- · pushed 2026-10-17; PyTorch ecosystem landscape member; NVIDIA/AMD hardware support
- · manual review: announced May 2026 — thin production hardening history; throughput headlines are vendor-reported
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.