TensorRT-LLM
NVIDIA's high-performance LLM inference engine.
About
TensorRT-LLM is NVIDIA's library for squeezing maximum throughput out of LLMs on NVIDIA GPUs: TensorRT kernels, in-flight batching, FP8/FP4 quantization, speculative decoding, and disaggregated prefill/decode serving. 14.8k stars, Apache-2.0, 48 releases in 12 months. It's the performance ceiling for NVIDIA inference — at the cost of an NVIDIA-only, compile-heavy workflow versus vLLM's ease. Use it when tokens-per-second-per-dollar is the metric that matters.
What it can do
- **In-flight batching**: maximum GPU utilization
- **Quantization**: FP8, FP4, AWQ, GPTQ support
- **Speculative decoding**: draft-model acceleration
- **Disaggregated serving**: separate prefill/decode
- **Multi-GPU/node**: tensor + pipeline parallelism
Setup
```bash
# Docker is the sane path; see the repo's quickstart
pip install tensorrt_llm # then build engines per the docs
```
Engine building (compiling the model) is a required step. Docs: nvidia.github.io/TensorRT-LLM.
Prerequisites
NVIDIA GPU (Ampere+ for FP8, Blackwell for FP4). Docker recommended.
Use cases
Works with
Trust score — 9/10
Scored 9/10 (verified tier, metrics-v1). Evidence: 14,802 GitHub stars (log-scaled vs catalog peers); last push 0d ago; 48 releases/12mo; 1640 open issues. Manual review: NVIDIA's LLM inference optimizer (in-flight batching, FP8/FP4, speculative decoding), 14.8k stars, 48 releases/12mo; NVIDIA-only, compile-heavy vs vLLM. License: Apache-2.0.
★ 14,802 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 7.9/10
- Maintenance · recency, releases, issue load
- 8.2/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 9.0/10
- · 14,802 GitHub stars (log-scaled vs catalog peers)
- · last push 0d ago; 48 releases/12mo; 1640 open issues
- · manual review: NVIDIA's LLM inference optimizer (in-flight batching, FP8/FP4, speculative decoding), 14.8k stars, 48 releases/12mo; NVIDIA-only, compile-heavy vs vLLM
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.