Where a type-safe Rust runtime earns its keep.
UniLLM is for engineers who want an inference runtime they can read, own, and extend. These are the workloads it is built for — all CPU today, with GPU on the roadmap.
Inference under your own Rust service
Put the Model trait and the inference engine directly inside your service instead of shelling out to a Python server. You own the process, the memory profile, and the error handling — no FFI boundary, no separate runtime to supervise.
- — Embed generate() in an existing Rust binary
- — One dependency, ahead-of-time compiled
- — Typed inputs and outputs across the call
Reading and tuning kernels
Performance engineers who want to see and change the math get SIMD kernels (AVX2, AVX-512, NEON) for quantized matmul, RMSNorm, RoPE, and SwiGLU in plain Rust — not buried behind a Python wrapper around C++.
- — Profile and tune the hot paths
- — Functional ops_fn surface reads as plain Rust
- — No opaque native layer in the way
Adding a new architecture
Because every model implements the same Model trait through the model_config! macro, bringing up a new architecture is mostly writing its forward pass. The 47 existing implementations are worked examples.
- — model_config! generates the boilerplate
- — Format-agnostic weights via WeightLoaderCore
- — Unit-test with dummy tensors, then real weights
Format-agnostic local inference
When your weights arrive as SafeTensors from one source, GGUF from another, and a PyTorch checkpoint from a third, auto_detect loads all three into the same ModelWeights container so the rest of your code stays identical.
- — SafeTensors, GGUF, and PyTorch under one loader
- — GGUF Q4_0 / Q8_0 dequantized at load time
- — auto_detect keeps call sites uniform
Where UniLLM is not the right tool
If you need GPU serving today, an OpenAI-compatible HTTP server, or token streaming to clients, UniLLM does not do those yet. It is a CPU-first runtime you build on, not a turnkey server. See the comparisons for how it stacks against other Rust runtimes.