Three cores, one runtime.
UniLLM is a small Cargo workspace whose crates split along clear lines. Almost everything is built out of three small traits and the types they carry.
Four crates, clear seams
crates/runtime
Core inference runtime: tensor ops, the Model trait, weight loading, and the 47 architecture implementations.
crates/inference
The high-level inference engine and batching.
crates/kv
The hybrid KV cache (RadixAttention + PagedAttention).
crates/scheduler
Request scheduling with continuous batching and chunked prefill.
Small traits that keep 47 architectures honest
TensorCore
A single Tensor backed by Arc<dyn TensorStorage>, with a Device enum for CPU, CUDA(usize), and Metal(usize). Ops go through TensorOps and a functional ops_fn module.
ModelCore
One Model trait — new, from_weights, forward, generate, to_device. Typed ModelInputs / ModelOutputs enums say which adapter you need for which use case.
WeightLoaderCore
from_safetensors, from_gguf, from_pytorch, and auto_detect produce a unified ModelWeights. GGUF weights are dequantized to f32 at load time; direct quantized inference is on the roadmap.
From weights to tokens
WeightLoader::auto_detect reads a SafeTensors, GGUF, or PyTorch file into a unified ModelWeights container.
Model::from_weights builds a concrete architecture that implements the Model trait.
Text is tokenized (GGUF tokenizers with byte-level fallback, or HuggingFace tokenizers).
forward() runs the pass through TensorOps; SIMD kernels handle quantized matmul, RMSNorm, RoPE, and SwiGLU.
A sampler (greedy, temperature, or top-p) picks the next token.
generate() loops until the stop condition and returns the decoded text.
Where it stops today
Inference runs on CPU. The hybrid KV cache and the scheduler are implemented and tested but not yet integrated into the autoregressive loop, and there is no HTTP server or SSE streaming yet. The features page lists exactly what ships and what does not.