Aleph Alpha has released Kolibri-1, a mixture-of-experts language model with 78 billion parameters, of which only about 3.5 billion are active per token. That ratio makes it a candidate for a laptop, yet Aleph Alpha’s own way to run it is a plugin for vLLM, an inference server built for data-centre GPUs. kolibri-llama-cpp brings Kolibri to llama.cpp instead. The goal is a quantized build that runs on a Mac with 48 GB of unified memory and gives the same answers as Aleph Alpha’s reference. Quantized builds already run there, at up to 64 tokens per second.
A small delta on a known model
Aleph Alpha’s vLLM plugin builds Kolibri on top of vLLM’s Qwen3-MoE classes and implements only what differs, and the port does the same in llama.cpp. The changes are kept as patches against a pinned llama.cpp commit and as commits on a fork branch, so that they can go upstream later.
The differences are small, but getting any one of them wrong changes the output. Four out of five layers attend to a sliding window of the 512 previous tokens plus the current one; every fifth layer sees the whole context. Positions are encoded (RoPE) only in the sliding layers. Every layer has norms both before and after attention and the experts. And the router picks 6 of 384 experts after adding a learned bias to the raw scores, but weights them without it. llama.cpp’s existing code for DeepSeek-style routers adds the bias after the sigmoid instead. Since the bias is not zero, that would pick different experts, so Kolibri got its own branch in the router. Its five operations per layer are cheap: on Metal they take at most 7.0% of a generated token.
Testing without the model
The BF16 checkpoint is 156 GB, the FP8 one 78 GB, and running Aleph Alpha’s reference on either needs hardware I don’t have yet. So rather than comparing the full model, the project compares every single step on tiny synthetic checkpoints. A Go tool writes random checkpoints in Kolibri’s shapes: with the real 384 experts and Top-6 router, with the real 48 query and 4 KV heads and the 513-token window, or with all 50 layers in their real pattern. Python checks run them through libllama on the CPU and on Metal and compare every intermediate tensor with a float64 recomputation built on the reference’s own functions.
On the CPU, the router and expert outputs stay within a normalized error of 4e-14 and the attention within 2e-13, and the full-attention layers’ cache holds every position up to the real context of 262,144 tokens. To show that the checks would actually catch a mistake, each one also runs against deliberately broken builds: a window one token too short or too long, RoPE on every layer, a missing Q or K norm. All of them are caught.
The tokenizer was already there
Kolibri has its own 128,000-token vocabulary, but it needed no new tokenizer code: llama.cpp’s byte-level BPE with the Qwen2 pre-tokenizer reproduces it exactly. 85 hand-picked cases and about 1.35 million fuzzed strings give the same token IDs and bytes as Hugging Face’s reference, and so does a fixed corpus of 10,673 English, German and code cases with 1,271,044 tokens. Passing it is not a given: with the default pre-tokenizer instead of Kolibri’s, only 129 of the 4,212 wikitext cases match. There was one trap. The tokenizer file skips two IDs, and the library ignores the IDs in the file anyway, so at runtime 75 reserved tokens sit two places lower than the file says. llama.cpp’s converter happens to do the same, and a test now pins it.
The chat template matches as well. For seven conversations with tool calls and reasoning, and for 31 ways of switching reasoning on or off, llama-server renders the same prompt as the reference, byte for byte. Only one combination, reasoning_effort: "none" next to enable_thinking: true, produced a thinking prompt where the reference turns thinking off; a one-line fix in llama-server’s shared code took care of it. Continuing a half-written assistant answer renders the same too, except when the reasoning itself is continued: llama.cpp leaves the think block open, the reference template closes it. Parsing such a continuation, llama.cpp even avoids a known gap in vLLM, which files a continued answer as reasoning when thinking is on.
On a 48 GB Mac
Then came the real checkpoint, on a Mac with an M5 Pro and 48 GB. The 156 GB of BF16 weights convert to a GGUF in under eight minutes, with a peak memory footprint of 7.5 GiB, because the converter streams one layer’s experts at a time. A check then compares all 903 tensors of the result with the source files, and every one is bit-exact. Unquantized, the full 50-layer model does run, but only on the CPU, at 1.45 tokens per second, with its experts read from the SSD.
To make it fit into memory, the routed experts, which hold 96.7% of the parameters, are quantized to 3 bits (Q3_K). Attention, the shared expert, the embeddings and the output stay at 8 bits, and the router and the norms at full float32. That file runs entirely on Metal:
| GGUF | Size | Runs on | Generation | Mean KLD | Same top token |
|---|---|---|---|---|---|
| BF16 | 149,065 MiB | CPU | 1.45 tokens/s | (baseline) | (baseline) |
| Q8_0 | 79,279 MiB | CPU | – | 0.050 | 92.5% |
| Q3_K experts, rest Q8_0 | 33,717 MiB | Metal | 60.7 tokens/s | 0.108 | 88.4% |
| IQ3_XXS gate/up, IQ4_XS down, imatrix, rest Q8_0 | 33,904 MiB | Metal | 64.0 tokens/s | 0.095 | 88.5% |
The last row needed an importance matrix, which tells the quantizer which weights matter most. English wikitext alone was not enough for a model with 384 experts per layer: in a test run, 12 to 28% of the experts in the last four layers never saw a single token, and an expert without data is quantized as if there were no matrix at all. The calibration text is therefore 45% English, 35% German Wikipedia and 20% source code, 128,098 tokens in all. The matrix changes nothing for Q3_K, but it makes the IQ types usable. With 3-bit IQ3_XXS for the gate and up projections and 4-bit IQ4_XS for the down projection, the divergence drops 12% below the Q3_K mix for 187 MiB more, and generation gets 5% faster. IQ3_XXS for all experts matches the Q3_K mix in quality at 30,342 MiB, which leaves room for longer contexts.
With a 32k context, the IQ3_XXS/IQ4_XS file takes 34,572 MiB of GPU memory, which leaves 3.4 GiB below macOS’s default Metal limit. The hybrid attention is what makes room for more: only the 10 full-attention layers grow with the context, by 20 KiB per token, while the 40 sliding layers stay at 100 MiB however long it gets. If I keep at least 2 GiB free, the file takes 64k tokens of context, or 128k once the KV cache is quantized to q8_0. Whether q8_0 costs quality is not measured yet. IQ3_XXS for all experts even reaches the model’s full 262,144 tokens with q8_0.
Through llama-server, the Q3_K mix answers a German question after reasoning first, skips the thinking when asked, and calls a weather tool with {"city": "Berlin"}. Longer contexts cost speed: with the cache filled to 32k, the IQ3_XXS/IQ4_XS file generates 41.2 tokens per second instead of 61.0. A run of 32,000 tokens through llama-server finished without errors at 47.56 tokens per second, falling from 60.4 to 39.4 along the same curve as the cache filled up. Memory stayed flat, but the model alone puts the 48 GB Mac at macOS’s “warn” memory pressure level.
The two large files can now use the GPU as well. With --cpu-moe, llama.cpp keeps the routed experts on the CPU, reads them on demand from the memory-mapped file, and runs attention, the router and the shared expert on Metal. On the Q8_0 and BF16 files, that used to crash with SIGBUS: llama.cpp gave each backend a single buffer from its first tensor in the file to its last, and since the Metal tensors sit in all 50 layers, that buffer covered nearly the whole file, far beyond Metal’s 37 GiB working set. A generic patch now maps only the ranges that hold a backend’s tensors, for Q8_0 2,447 MiB in 152 buffers. Both files now run, and the model that fits keeps its speed (63.7 against 64.0 tokens per second). The large files don’t get any faster, though:
In every streaming setup, the CPU still reads and multiplies the routed experts, 96.7% of the weights. Moving the rest to Metal saves little and adds two handoffs between CPU and GPU per layer. Where streaming shines is memory: on the CPU alone, the Q8_0 process itself stays at about 2.5 GB, and Metal maps only 2,447 MiB (Q8_0) or 4,440 MiB (BF16), against 33,572 MiB for the model that fits. The rest of the file lives in the OS page cache, which macOS can drop at any time. Streaming costs speed even when nothing comes from the disk: with all experts of the IQ3_XXS/IQ4_XS file on the CPU, generation drops from 62.85 to 18.06 tokens per second, although the page cache serves practically every expert read, because the 33 GB file fits in RAM.
Would it fit a MacBook Pro with 24 GB? I only have the 48 GB machine, so I can only estimate it. Fully on Metal, no: even the smallest variant, IQ3_XXS for all experts, is 30,342 MiB, more than the whole memory. Streamed, it depends on how often the same experts come back. To find out, I recorded which experts four fixed chat sessions select in every layer: coding, research, HR tool use and EU medical-device regulation, 10,645 to 28,450 tokens each. There is no small working set: 66 to 73% of a layer’s experts are used within the first 1,000 tokens. But their use is uneven, so a cache helps. Without one, a token needs 510 MB of experts; replayed through a simulated LRU cache of 128 experts per layer, 10.88 GB, 90.1 to 94.8% of the reads hit, and a token loads only 27 to 50 MB. llama.cpp has no expert cache of its own, so on 24 GB that job falls to the page cache; how well it does it, and how fast the SSD delivers the rest, I haven’t measured.
Checking against the reference code
The quality numbers in the table compare llama.cpp with llama.cpp, the quantized files against the port’s own BF16 model, so they can’t catch a bug in the port itself. And two results did look suspicious. For dense models, 8-bit quantization usually stays near a divergence of 0.001 with over 99% of top tokens unchanged; here Q8_0 reaches 0.050 and 92.5%, about half the divergence of every 3-bit variant. Raw German completions without the chat template also fall into repetition, even in the unquantized model: “Die Hauptstadt von Deutschland ist” continues with ” Deutschland Deutschland Deutschland …”, while English prompts and German through the chat template stay coherent.
Running Aleph Alpha’s vLLM reference still needs hardware I don’t have, so the next best thing is its model code. The project now has a whole-model forward pass in torch, ported from the vLLM plugin and the Qwen3-MoE classes it builds on. It reads the weights from the BF16 GGUF and only touches the experts each token is routed to. On the tiny fixture it matches libllama to a normalized error below 1e-12, and two deliberately broken versions of it fail as they should. On the real weights, the comparison takes 15 minutes with a peak footprint of 11 GB.
Against the float32 reference, llama.cpp’s BF16 model reaches a divergence of 0.0334, with 96.5% of top tokens the same. Rounding the reference itself to BF16, the precision vLLM runs at, costs 0.0327 and 93.7%. In other words, the port is as close to the reference as the reference is to itself in BF16. The layer outputs drift apart smoothly, from a normalized error of 1e-7 in layer 0 to at most 0.08, without any single step standing out. The drift comes from the router: for 22.3% of tokens and layers, the 6th and 7th expert scores lie within 0.01 of each other, so the smallest rounding difference swaps an expert. By the last layer, only 64% of tokens use the same experts on both sides.
Two later checks take a closer look. Node by node, the token embeddings match exactly, and in the first sliding and the first full attention layer every step stays within a normalized error of 6.2e-6 of the float32 reference, 5 to 72 times below what BF16 rounding costs the reference itself. A router probe over 25,600 token-layer pairs then finds that llama.cpp on the CPU picks the same six experts as the float32 reference for 84.05% of them, the BF16-rounded reference for only 75.88%. 97% of llama.cpp’s swaps happen where the 6th and 7th score are less than 0.1 apart. On Metal, llama.cpp comes even closer (a divergence of 0.0178, 87.17% same sets); the run doesn’t show why.
That answers the German question: the reference, too, continues the prompt with ” Deutschland Deutschland Deutschland …”, in float32 and in bfloat16, and llama.cpp picks the same greedy token at all 16 positions. The repetition is what Kolibri’s model code computes on these weights, not a llama.cpp bug. The near-ties also make the router a plausible cause of the Q8_0 divergence, since BF16 rounding alone already costs about as much. The reference only ran the BF16 weights, though, so an additional Q8_0 defect is not ruled out yet. Nor is a misreading of the vLLM code that the port and the torch version would share; only vLLM itself can rule that out. The tooling for that run is ready. On the tiny checkpoint, vLLM 0.29.0 with Aleph Alpha’s plugin runs on its macOS CPU backend and agrees with libllama to a normalized logit error of 1.70e-12, with the same experts everywhere. The real weights need a GPU with more than 156 GB of memory.
A stored reference for later llama.cpp versions
The comparison above recomputes the torch reference on every run, for two inputs shorter than the sliding window. So the reference is now stored once, for six fixed cases from raw German, English and code prompts to 1,024 tokens of German text, which took 2 h 23 min. From then on, a single command checks the tokenizer corpus, the tiny checkpoints and these six cases in 27 minutes. It compares all 451 intermediate tensors and the logits of each case with a recorded llama.cpp run, bit for bit, and names the first tensor that changed; a changed RMS-norm epsilon, for example, fails right at the Q norm of layer 0. That makes it the check to run after every upstream llama.cpp change.
The stored cases also settled two questions. The long German case is the only one in which the sliding layers actually drop keys, and there llama.cpp is closer to the float32 reference (a divergence of 0.0463) than the reference’s own bfloat16 run (0.1025), so nothing points at the window mask. The short prompts, on the other hand, show how sensitive this model is to BF16 rounding: on the code prompt, the bfloat16 reference ends up at 0.2868 from float32, llama.cpp at 0.0026.
Trying it
The main thing left is that vLLM run on the real weights, along with the Q8_0 question and the recommended sampling settings. The Metal fix for --cpu-moe is generic llama.cpp code: current llama.cpp master still crashes when the mapped file exceeds Metal’s working set, the patch applies there on its own, and two earlier upstream reports of the crash were closed as not planned. llama.cpp’s contribution rules don’t accept AI-written bug reports or pull requests, so mine are still to be written by hand. To try it now, you need about 312 GB of free disk for the conversion and a Mac with 48 GB for the quantized model. The README describes how to apply the patches, build llama.cpp and run the checks; if you’d rather build from a branch, the same changes are commits on feat/kolibri in my llama.cpp fork. This is an independent project, not affiliated with Aleph Alpha.