BentoML
Turn any model inference script into a production REST API.
About
BentoML is a unified model-serving framework: decorate a Python class and your inference script becomes a production REST API server, with Docker images generated from a simple config, dynamic batching, model parallelism, and multi-model inference-graph orchestration. It supports any ML framework and modality (not just LLMs), runs locally for dev/debug, and deploys to any Docker environment or the managed BentoCloud. Apache-2.0, 8.9k stars. For pure LLM token throughput, compare against vLLM/TGI/LMDeploy; choose BentoML for heterogeneous multi-model services.
What it can do
- **Service decorator**: inference code → REST API in a few lines
- **Docker made simple**: reproducible images from a config file
- **Serving optimizations**: dynamic batching, model parallelism, multi-stage pipelines
- **Multi-model graphs**: orchestrate heterogeneous model compositions
- **Any framework**: PyTorch, Transformers, XGBoost, custom code — any modality
- **BentoCloud**: managed production deployment (paid)
Setup
```bash
pip install -U bentoml # Python 3.9+
```
```python
# service.py
import bentoml
@bentoml.service
class Summarization:
@bentoml.api
def summarize(self, text: str) -> str:
...
```
```bash
bentoml serve service.py # local server; bentoml build for Docker
```
Docs: docs.bentoml.com.
Prerequisites
Python 3.9+. Docker for containerized deploys (optional for local serve).
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: 8,899 GitHub stars (log-scaled vs catalog peers); blended with ~134,418 monthly package downloads; last push 7d ago; 13 releases/12mo; 228 open issues. Manual review: unified model serving (REST APIs, Docker, dynamic batching); 8.9k stars; vLLM/TGI beat it on pure LLM throughput; BentoCloud is the monetization path. License: Apache-2.0.
★ 8,899 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 7.5/10
- Maintenance · recency, releases, issue load
- 8.5/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · 8,899 GitHub stars (log-scaled vs catalog peers)
- · blended with ~134,418 monthly package downloads
- · last push 7d ago; 13 releases/12mo; 228 open issues
- · manual review: unified model serving (REST APIs, Docker, dynamic batching); 8.9k stars; vLLM/TGI beat it on pure LLM throughput; BentoCloud is the monetization path
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.