LiteLLM
One OpenAI-compatible endpoint for 100+ LLM providers
About
LiteLLM is a self-hosted LLM gateway that gives every application a single OpenAI-compatible endpoint in front of 100+ providers (OpenAI, Anthropic, Bedrock, Azure, Ollama, vLLM, TogetherAI, and more — provider-claimed). Requests go in OpenAI format and come back in OpenAI format, so no client code changes are needed.
The proxy layer adds virtual API keys, spend tracking and budgets, load balancing across models and deployments, and an Admin UI. There is also a Python SDK for calling 100+ LLMs from a single app without the gateway. Core is MIT; features under enterprise/ require a commercial license.
What it can do
- • Unified OpenAI-compatible gateway: /chat/completions, /completions, /embeddings, /models across 100+ providers (provider-claimed count)
- • Virtual API keys with per-key budgets, rate limits, and spend tracking per user/team
- • Load balancing and fallbacks across models and deployments; reported 1.5k+ req/s in provider load tests (provider-claimed)
- • Guardrails (prompt injection/PII filtering) and request logging to one location
- • Admin UI for key management, usage dashboards, and configuration
- • MCP tools and A2A agent support behind the same endpoint (per official docs)
- • Python SDK alternative (litellm.completion) for single-app use without the gateway
- • Works with OpenAI, Anthropic, Mistral SDKs, LangChain, and LlamaIndex clients
Setup
- Install the proxy extra (needs Python 3.10+; uv provisions it automatically): uv tool install 'litellm[proxy]'
- Start the gateway for a provider model, e.g.: litellm --model openai/gpt-5.6-luna (set the provider key first, e.g. export OPENAI_API_KEY=...)
- The proxy listens at http://0.0.0.0:4000 — confirm with: litellm --test
- For production use, write a config.yaml with a model_list (model aliases, api_base, api_key, fallbacks) and run: litellm --config your_config.yaml
- Point any OpenAI client at base_url="http://localhost:4000" and call /chat/completions, /completions, /embeddings, or /models
- Open the Admin UI in a browser to manage virtual keys, budgets, and spend tracking; secure the deployment (master key + auth on /metrics) before exposing it beyond localhost
Prerequisites
Python 3.10+; API keys for whichever provider models you route to (OpenAI, Anthropic, Bedrock creds, etc.); optional Postgres/Redis for production state
Use cases
Works with
Trust score — 9/10
Scored 9/10 (verified tier, metrics-v1). Evidence: 61,075 GitHub stars (log-scaled vs catalog peers); blended with ~92,239,101 monthly package downloads; last push 0d ago; 100 releases/12mo; 5472 open issues. Manual review: unverified /metrics PII-exposure report (#24530); opt-out telemetry; '100+ providers' provider-claimed. License: MIT core + commercial Enterprise tier (dual).
★ 61,075 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 9.2/10
- Maintenance · recency, releases, issue load
- 8.2/10
- Openness · license + docs
- 8.6/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · 61,075 GitHub stars (log-scaled vs catalog peers)
- · blended with ~92,239,101 monthly package downloads
- · last push 0d ago; 100 releases/12mo; 5472 open issues
- · manual review: unverified /metrics PII-exposure report (#24530); opt-out telemetry; '100+ providers' provider-claimed
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.