ExLlamaV2
Fast local LLM inference library with hand-written CUDA kernels — 50-85% faster than llama.cpp on NVIDIA GPUs.
About
ExLlamaV2 is a speed-focused inference library by solo developer turboderp for running quantized LLMs locally on consumer NVIDIA GPUs. It uses the EXL2 format — a quantization scheme where different layers get different bit precisions based on sensitivity to quantization error, with any average bitrate from 2 to 8 bits per weight — plus hand-written CUDA kernels optimized for NVIDIA tensor cores. Benchmarks show 50-85% faster token generation than llama.cpp at the same quantization level.
The tradeoffs are explicit: NVIDIA GPUs only, models must fit entirely in VRAM, and there is no Ollama or LM Studio support — access is through TabbyAPI (the official recommended OpenAI-compatible server) or text-generation-webui. The v0.1.0+ dynamic generator adds paged attention via Flash Attention, dynamic batching, smart prompt caching, and speculative decoding. Note that development has moved on to ExLlamaV3, so ExLlamaV2 is in maintenance while remaining the fastest option for its supported hardware.
What it can do
- EXL2 mixed-precision quantization (2-8 bits per weight)
- Dynamic generator with batching, prompt caching, and speculative decoding
- GPTQ and FP16 model support alongside EXL2
- OpenAI-compatible serving via TabbyAPI
Setup
pip install the exllamav2 package from GitHub (turboderp-org/exllamav2), download an EXL2/GPTQ quantized model from Hugging Face, and serve it via TabbyAPI for an OpenAI-compatible endpoint.
Prerequisites
An NVIDIA GPU with enough VRAM for the model; Python 3; CUDA toolkit.
Use cases
Works with
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 64; Trending score 72/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.