TabbyAPI
OpenAI-compatible API server for ExLlamaV2/V3 local inference.
About
TabbyAPI is the official API backend server for ExLlamaV2 and ExLlamaV3, the fast inference libraries for running LLMs locally on consumer GPUs. It is a FastAPI-based serving engine focused on high-quality quantized model serving (EXL2, EXL3, GPTQ, FP16) with continuous batching via paged attention.
It exposes an OpenAI-compatible API (chat completions, completions, embeddings, model listing) plus KoboldAI API compatibility and extensive management endpoints. Features include HuggingFace model downloading, embedding model support, JSON schema/regex/EBNF constrained generation, speculative decoding via draft models, multi-LoRA with independent scaling, and OAI-style tool/function calling. The project notes it is a rolling-release hobby project not intended for production servers.
What it can do
- OpenAI-compatible chat, completions, and embeddings endpoints
- EXL2, EXL3, GPTQ, and FP16 quantized model serving
- Continuous batching with paged attention
- Constrained generation (JSON schema, regex, EBNF)
- Speculative decoding and multi-LoRA support
Setup
Clone the repo, install dependencies, and run the server (default port 5000); point any OpenAI-compatible client at the local endpoint. See the project wiki for the step-by-step guide.
Prerequisites
An NVIDIA GPU (Ampere or newer for paged attention), Python, and downloaded ExLlama-compatible model weights.
Use cases
Works with
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 107; Trending score 74/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.