ONNX Runtime Generative AI
Microsoft's generate() API for running LLMs on-device across CPU, GPU, and NPUs.
About
ONNX Runtime Generative AI (onnxruntime-genai) is Microsoft's extension that adds the full generative-AI loop on top of ONNX Runtime: preprocessing and postprocessing, inference, logits processing, sampling strategies, and KV-cache management behind a simple generate() call that can stream tokens one at a time. It supports Llama, Phi, Gemma, Mistral, Qwen, and related families.
It ships APIs for Python, C#, C/C++, and Java, runs on Linux, Windows, macOS, and Android, and accelerates through CUDA and DirectML today with Qualcomm QNN, ROCm, and NPU targets in progress. It is the engine behind on-device inference on Windows Copilot+ PCs and a pragmatic pick when one codebase must reach many chips.
Honest limitations: models must live in the ONNX format, so novel architectures sometimes wait on opset coverage or custom ops before they export cleanly. Tokenizer, chat template, and sampling config ride alongside as sidecar files rather than inside the .onnx graph, which adds packaging friction versus formats like GGUF.
What it can do
- generate() API with streaming token output and KV-cache management
- Python, C#, C/C++, and Java APIs across desktop, mobile, and edge
- CUDA and DirectML acceleration with QNN, ROCm, and NPU support in flight
Setup
Install via pip, NuGet, or build from source; convert or download an ONNX-format model and call generate().
Prerequisites
An ONNX-format LLM plus tokenizer files; platform SDKs for the target OS and accelerator.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 81; Trending score 84/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.