TorchServe

TorchServe is the official open-source model serving framework for PyTorch, developed by AWS and Meta. Models are packaged into .mar archives with custom Python handlers for preprocessing and inference logic, then served over REST or gRPC with dynamic batching, multi-model serving, model versioning for A/B testing, Prometheus metrics, and worker auto-scaling. It is preinstalled in AWS PyTorch Deep Learning Containers and used under the hood by services like SageMaker and Vertex AI endpoints.

The honest caveats: the project is in limited maintenance mode — community guidance flags checking environment compatibility before adopting. It predates the LLM era, so serving large LLMs needs the vLLM-enabled path rather than the classic handler model. For modern LLM serving, dedicated engines like vLLM or TensorRT-LLM are usually better; TorchServe remains strongest for traditional PyTorch vision and tabular models.

pip install torchserve torch-model-archiver, package your model as a .mar archive, then torchserve --start.