Triton Inference Server
NVIDIA's production inference server for any framework.
About
Triton Inference Server is NVIDIA's open-source serving platform: deploy models from TensorRT, PyTorch, ONNX, OpenVINO, or plain Python behind one HTTP/gRPC API, with dynamic batching, concurrent model execution, model ensembles (pre/post-processing pipelines), and metrics. 11k stars, BSD-3-Clause. It's the enterprise workhorse for GPU inference fleets — strongest on NVIDIA hardware, naturally. For pure LLM text generation, vLLM is usually simpler; for heterogeneous model zoos (vision + NLP + recommenders on shared GPUs), Triton is the standard.
What it can do
- **Multi-backend**: TensorRT, PyTorch, ONNX, OpenVINO, Python, custom
- **Dynamic batching**: automatic request batching for throughput
- **Ensembles**: chain models + pre/post-processing into pipelines
- **Concurrent models**: many models sharing one GPU fleet
- **Standard APIs**: HTTP/REST and gRPC with Prometheus metrics
Setup
```bash
docker run --gpus all -p8000:8000 -v /models:/models nvcr.io/nvidia/tritonserver:25.10-py3 tritonserver --model-repository=/models
```
Models go in a repository layout (config.pbtxt + weights). Docs: github.com/triton-inference-server/server.
Prerequisites
NVIDIA GPU + drivers (best experience); Docker. Models in Triton repository format.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: 11,073 GitHub stars (log-scaled vs catalog peers); last push 0d ago; 12 releases/12mo; 900 open issues. Manual review: NVIDIA's production inference server (multi-backend, batching, ensembles); 11k stars; NVIDIA-centric; heavier than vLLM for pure-LLM serving. License: BSD-3-Clause.
★ 11,073 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 7.6/10
- Maintenance · recency, releases, issue load
- 8.2/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · 11,073 GitHub stars (log-scaled vs catalog peers)
- · last push 0d ago; 12 releases/12mo; 900 open issues
- · manual review: NVIDIA's production inference server (multi-backend, batching, ensembles); 11k stars; NVIDIA-centric; heavier than vLLM for pure-LLM serving
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.