vLLM
High-throughput LLM serving with PagedAttention
About
vLLM is the de-facto open-source engine for high-throughput LLM inference, built around PagedAttention for efficient KV-cache memory management. It runs offline batched inference via a Python API (LLM.generate / LLM.chat) and online serving as an OpenAI-compatible server (vllm serve) exposing /v1/chat/completions, /v1/completions, and /v1/models.
It supports a huge model catalog from Hugging Face, quantization formats (AWQ, GPTQ, FP8), LoRA, prefix caching, and hardware from NVIDIA CUDA to AMD ROCm, Intel XPU, Google TPU, and Apple Silicon. Independent 2026 benchmarks put it neck-and-neck with SGLang on unique-prompt workloads and slightly behind on prefix-heavy chat/RAG traffic.
What it can do
- • PagedAttention-based inference engine for high-throughput text generation
- • OpenAI-compatible server: drop-in replacement for the OpenAI API (chat, completions, embeddings, vision)
- • Offline batched inference via the Python LLM class (generate / chat methods)
- • Continuous batching, prefix caching, chunked prefill, and quantization (AWQ, GPTQ, FP8, Marlin)
- • LoRA adapter serving, speculative decoding, and structured/guided decoding (JSON schema)
- • Vision-language models, audio (Whisper-style) models, and reward/embedding models
- • Broad hardware support: NVIDIA CUDA, AMD ROCm, Intel XPU, Google TPU, Ascend NPU, Apple Silicon (via vLLM-Metal)
- • Prometheus metrics, request logging, and optional API-key auth (--api-key / VLLM_API_KEY)
Setup
- Check prerequisites: Linux (or Docker), Python 3.11–3.14, and a supported GPU (NVIDIA CUDA works out of the box via pip)
- Create a Python environment: uv venv --python 3.12 --seed && source .venv/bin/activate
- Install vLLM: uv pip install vllm --torch-backend=auto (then vllm download-kernels to precompile kernels and skip the first-startup compile wait)
- (Optional) Disable anonymous usage-stats telemetry before first run: export VLLM_NO_USAGE_STATS=1
- Serve a model: vllm serve Qwen/Qwen2.5-1.5B-Instruct — models download automatically from Hugging Face on first use
- Test it: curl http://localhost:8000/v1/models — then send requests in OpenAI format to http://localhost:8000/v1/chat/completions
- For production, protect the server with --api-key (or VLLM_API_KEY) since requests are unauthenticated by default
Prerequisites
Linux (best support; Docker for other setups), Python 3.11–3.14, a CUDA/ROCm/XPU GPU for real workloads (CPU works for tiny models only); Hugging Face access for model downloads
Use cases
Works with
Trust score — 9/10
Scored 9/10 (verified tier, metrics-v1). Evidence: 93,654 GitHub stars (log-scaled vs catalog peers); blended with ~1,857,414 monthly package downloads; last push 0d ago; 32 releases/12mo; 8713 open issues. Manual review: anonymous usage stats on by default (opt-out); no prompts sent. License: Apache-2.0.
★ 93,654 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 9.1/10
- Maintenance · recency, releases, issue load
- 8.2/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 9.0/10
- · 93,654 GitHub stars (log-scaled vs catalog peers)
- · blended with ~1,857,414 monthly package downloads
- · last push 0d ago; 32 releases/12mo; 8713 open issues
- · manual review: anonymous usage stats on by default (opt-out); no prompts sent
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.