Everything in the runtime.
UniLLM is a small Cargo workspace with clear seams. Here is what ships today — grouped by layer — and an honest list of what is still on the roadmap.
One trait, every format. WeightLoaderCore reads what you have and hands the model a unified container.
SafeTensors
Native SafeTensors loading via from_safetensors.
GGUF
GGUF with Q4_0 / Q8_0 dequantization to f32 at load time. Direct quantized inference is on the roadmap.
PyTorch
PyTorch checkpoint loading via from_pytorch.
auto_detect
Pass a path and let the loader pick the format; models never see the difference.
47 architectures implement one Model trait through the model_config! macro, so a new model is mostly its forward pass.
Core LLMs
LLaMA, Qwen, Gemma, Phi, DeepSeek, Mistral, Mixtral.
GPT family
GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, MPT.
MoE
DeepSeek-MoE, DBRX, Grok, Arctic, Jamba.
Linear attention
RWKV-4, RWKV-6, RecurrentGemma, Mamba.
Vision-Language
Qwen2-VL, Phi-3-Vision, InternVL, CogVLM, LLaVA, CLIP.
Audio / Speech
Whisper, Wav2Vec2, HuBERT, MusicGen, Encodec.
A device-agnostic Tensor with a unified ops surface and SIMD kernels for the hot paths.
Device enum
CPU, CUDA(idx), Metal(idx). Inference runs on CPU today; GPU wiring exists via Candle feature flags.
SIMD kernels
AVX2, AVX-512, and NEON implementations for quantized matmul, RMSNorm, RoPE, and SwiGLU.
Functional ops
An ops_fn module so model code reads as plain Rust functions, not chains on opaque types.
Typed I/O
ModelInputs / ModelOutputs enums cover text, image, multimodal, and audio.
The runtime ships the building blocks for high-throughput serving; integration into the generation loop is in progress.
Hybrid KV cache
RadixAttention + PagedAttention with an adaptive tiering policy. Implemented and tested; loop integration is next.
Scheduler
Continuous batching, chunked prefill, and admission control.
Sampling
Greedy, temperature, and top-p (nucleus) sampling.
Tokenization
GGUF tokenizers with byte-level fallback, plus HuggingFace tokenizers.
What UniLLM cannot do today
- GPU acceleration — inference runs on CPU today; CUDA / Metal wiring exists but is unoptimized.
- Real-weight validation beyond LLaMA — the other 46 architectures pass unit tests with dummy tensors.
- No HTTP server yet — inference is CLI or programmatic.
- No token streaming to clients (the CLI prints tokens; SSE is not wired up).
- No repetition penalty, beam search, or min-p sampling.
- KV cache is not yet integrated into the autoregressive loop.