Straight answers.
What UniLLM does today, what it does not, and where the roadmap begins. We do not overstate the inference story.
What is UniLLM?
UniLLM is a modular LLM inference runtime written in Rust. It provides a unified, type-safe interface for running large language models across 47 architectures through three composable abstractions: TensorCore, ModelCore, and WeightLoaderCore.
Does UniLLM run on GPU?
Not yet. Inference runs on CPU today. The tensor abstraction is wired for CUDA and Metal via Candle feature flags, but no GPU-specific optimization has been done. GPU acceleration is on the near-term roadmap.
Which weight formats does it load?
WeightLoaderCore reads SafeTensors, GGUF (with Q4_0 / Q8_0 dequantization to f32 at load time), and PyTorch files. auto_detect picks the format so models never need to know about it.
Are all 47 architectures production-ready?
No. LLaMA is validated end-to-end with real GGUF weights. The other 46 architectures have correct forward-pass implementations covered by unit tests with dummy tensors and are being progressively validated against real weights.
Which models are implemented?
Core LLMs (LLaMA, Qwen, Gemma, Phi, DeepSeek, Mistral, Mixtral), the GPT family (GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, MPT), MoE models (DeepSeek-MoE, DBRX, Grok, Arctic, Jamba), linear-attention models (RWKV-4/6, RecurrentGemma, Mamba), vision-language models (Qwen2-VL, Phi-3-Vision, InternVL, CogVLM, LLaVA, CLIP), and audio models (Whisper, Wav2Vec2, HuBERT, MusicGen, Encodec).
Is there an HTTP server?
Not yet. Inference is CLI or programmatic through the Rust API. Token streaming to clients over SSE is not wired up either — the CLI prints tokens as they are produced.
What sampling strategies are supported?
Greedy, temperature, and top-p (nucleus) sampling today. Repetition penalty, beam search, and min-p sampling are not yet implemented.
How does the KV cache work?
UniLLM implements a hybrid RadixAttention + PagedAttention KV cache with an adaptive tiering policy. It is implemented and tested but not yet integrated into the autoregressive generation loop.
Who is UniLLM for?
Rust engineers building LLM infrastructure who want to put an inference runtime under their own service rather than wrap someone else’s Python server, and performance engineers who want to read and tune kernels in a typed, ahead-of-time-compiled language.
What license is UniLLM under?
Apache-2.0. The source and LICENSE live at github.com/cognisoc/unillm.
Still have a question about the runtime, a model, or weight loading? Get in touch or open an issue.