---

license: apache-2.0

base_model: Aleph-Alpha/Kolibri-1

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

language:

- de

- en

tags:

- gguf

- moe

- apex

- quantized

- imatrix

- kolibri

- reasoning

- tool-calling

- llama.cpp

---

Kolibri-1 — APEX GGUF

GGUF quantizations of Aleph-Alpha/Kolibri-1 —

Aleph Alpha's German/English reasoning model: a 78 B-total / 3.46 B-active mixture-of-experts

trained from scratch, with tool calling, an explicit reasoning mode and a 262 K native context.

Apache-2.0.

Because only \~3.5 B parameters are active per token, it decodes like a small model while carrying

the knowledge capacity of a large one: \~51 tokens/s on a DGX Spark (GB10) at 4.6 bpw.

> ### ⚠️ Requires a forked llama.cpp — upstream CANNOT load this model

> Stock llama.cpp will fail to load these files at the architecture check. As of 2026-10-07

> upstream has no support for the kolibri1 architecture (0 hits at origin/master

> b86d2f075).

>

> You need this fork and branch:

> ### 👉 brywil/llama.cpp — branch kolibri1

>

> ```bash

> git clone -b kolibri1 https://github.com/brywil/llama.cpp llama.cpp-kolibri

> cd llama.cpp-kolibri

> cmake -B build -DGGML_CUDA=ON # drop -DGGML_CUDA=ON for a CPU-only build

> cmake --build build -j --config Release

> ```

>

> Verified at commit 68fa28b56, two commits on top of upstream 22397c31a. They add the

> architecture (converter, tensor loading, compute graph), one new MoE gating mode, and accept

> kolibri1 as a tokenizer pre-type name; nothing else is changed. This fork is maintained as the home for Kolibri support — it is not a staging

> branch for an upstream PR.

>

> Every llama-server / llama-cli invocation on this page means the binary from that build

> tree.

Architecture

arch kolibri1 78.1 B params, vocab 128,000

layers 50

FFN MoE 384 experts, top-6, +1 shared (ungated); expert ffn 512

routing select top-6 on logits + per-expert bias, weight by sigmoid(logits), no renorm

attention 48 heads / 4 KV, head_dim 128, per-head q/k RMSNorm

4 sliding-window layers (window 513, RoPE) : 1 full-attention layer (no RoPE)

norms sandwich: pre- and post-norm around both attention and FFN

context 262,144 native

Two properties matter for running it:

• KV cache is cheap. Only the 10 full-attention layers grow with context — **\~20 KiB per

token at f16 (10 × 4 × 128 ÷ 256), so the full 262,144-token context costs about 5 GiB**.

The 40 sliding-window layers hold a small constant-size cache. Decode speed stays nearly flat

as context grows.

• Every K-quant is legal. The hidden size (2560) and expert width (512) are both divisible by

256, so no tensor is silently substituted with a block-32 type — the files contain exactly the

types listed below.

Files

| file | size | bpw | wiki KLD | wiki top-1 | code KLD | code top-1 |

|---|---:|---:|---:|---:|---:|---:|

| Kolibri-1-uniform-Q4_K.gguf | 41.89 GiB | 4.61 | 0.076 | 89.7% | 0.066 | 95.3% |

| Kolibri-1-APEX-i-quality.gguf | 39.91 GiB | 4.39 | 0.085 | 89.0% | 0.066 | 95.0% |

| Kolibri-1-uniform-IQ3_S.gguf | 32.55 GiB | 3.58 | 0.103 | 88.4% | 0.077 | 94.7% |

| Kolibri-1-APEX-i-compact.gguf | 31.67 GiB | 3.48 | 0.118 | 87.0% | 0.084 | 94.2% |

| Kolibri-1-APEX-i-mini.gguf | 25.30 GiB | 2.78 | 0.170 | 84.1% | 0.092 | 93.1% |

| Kolibri-1-APEX-2.4bpw.gguf | 22.70 GiB | 2.50 | 0.212 | 81.1% | 0.132 | 91.6% |

| imatrix-kolibri1-de_en_code-filled.gguf | 0.41 GiB | — | importance matrix used for every file | | | |

KLD and top-1 are measured against the BF16 model's own logits (mean KL divergence per token,

and how often the quant's top token matches BF16's), on 20 × 512 tokens each of WikiText-2 test and

a code-heavy corpus. Lower KLD / higher top-1 is closer to the original.

Which one to take

• uniform-Q4_K (41.89 GiB) — the default. Closest to BF16 on every measure; 42/42 on the

agentic suites below. Fits a 64 GB machine with the full 262 K context.

• uniform-IQ3_S (32.55 GiB) — the \~3.5 bpw pick. Fits 48 GB with plenty of context.
• APEX-i-mini (25.30 GiB) — for \~32 GB of memory, with a moderate context.
• APEX-2.4bpw (22.70 GiB) — the smallest, and **it also cleared every agentic coding task

(42/42, 253/253)** while decoding faster (\~57 t/s). Prose suffers first (81% top-1 on WikiText), but

code holds up far better (91.6% top-1) — true at every size here. For coding on a tight budget,

this is the one.

APEX-i-quality and APEX-i-compact use the edge-heavy APEX band recipe at roughly the same sizes

as the two uniform files. They measured slightly behind them on mean KLD and top-1, inside the

noise at \~4.4 bpw (see Limitations 2), and they keep a somewhat better code p99. APEX-i-quality is

also agentic-verified: 42/42 at both effort high and low on a 16 GB GPU (below). It is \~2 GB smaller

than uniform-Q4_K, which matters when experts spill to system RAM.

What is in each file

The expensive-to-get-wrong tensors are protected in every file, and apart from one exception

(below) only the routed experts change between files:

| tensors | type |

|---|---|

| routers, expert-selection bias, all norms | F32 |

| full-attention layers' q/k/v/output (10 layers) | Q8_0 |

| sliding-window layers' q/k/v/output (40 layers) | Q6_K |

| shared expert, output head | Q8_0 |

| token embedding | Q6_K |

The exception is APEX-2.4bpw, which to reach its size also drops 32 of the 40 sliding-window

layers' attention projections to Q3_K (8 stay at Q6_K; the full-attention layers stay at Q8_0).

Routed experts (96.7% of the parameters):

| file | experts |

|---|---|

| uniform-Q4_K | Q4_K, every layer |

| uniform-IQ3_S | IQ3_S, every layer |

| APEX-i-quality | Q6_K (first/last 5 layers) · Q4_K (next 5 in from each end) · Q3_K (middle 30) |

| APEX-i-compact | Q4_K (outer 20 layers) · IQ2_S (middle 30) |

| APEX-i-mini | Q3_K (outer 20 layers) · IQ2_XXS (middle 30) |

| APEX-2.4bpw | IQ3_XXS (edge 10) · IQ2_S (near 10) · IQ2_XXS (middle 30) |

Every file's tensor-type histogram was read back from the shipped bytes and checked against its

recipe.

The importance matrix was computed on a bilingual, code-inclusive corpus (332 × 512 tokens:

\~40% English prose, \~20% German, \~40% code). Experts that calibration never routes to (996 of 19,200,

60% of them in the first three layers) are given their layer's mean importance rather than a flat default.

Agentic coding — 42/42 tasks, 253/253 test cases, at 4.6, 4.4 and 2.5 bpw

The model drives a real tool-use loop (OpenCode: read, write and run files) and is graded by a

fresh pytest it never sees, at the model card's sampling (temp 1.0, top-p 0.97, top-k 128),

reasoning effort high, on a DGX Spark (GB10). Two files, identical settings:

| suite | uniform-Q4_K (41.89 GiB) | APEX-2.4bpw (22.70 GiB) |

|---|---|---|

| general (25) | 25/25 · 194,914 tok · 81 min | 25/25 · 193,296 tok · 70 min |

| hard (5) | 5/5 · 59/59 tests · 52,411 tok · 21 min | 5/5 · 59/59 tests · 48,763 tok · 17 min |

| frontier (9) | 9/9 · 142/142 tests · 111,342 tok · 43 min | 9/9 · 142/142 tests · 100,634 tok · 39 min |

| repair/mutation (3) | 3/3 · 27/27 tests · 16,490 tok · 7 min | 3/3 · 27/27 tests · 12,882 tok · 6 min |

| TOTAL | 42/42 · 253/253 · 375,157 tok · 2.5 h | 42/42 · 253/253 · 355,575 tok · 2.2 h |

Zero failures and zero truncated responses on both. Each task ran in a single uninterrupted

context — the agent's history was never summarised or compacted — with a 262,144-token window; the

largest context any request actually used was 58,529 tokens.

Half the bits cost nothing measurable here. The 2.5 bpw file scored identically and spent about

the same number of tokens (one run each — read that as a tie, not a saving); it finished sooner

because it decodes faster.

Reasoning effort low is worth trying. At --reasoning-effort low, uniform-Q4_K also scored

42/42 (253/253) with 303,509 output tokens and 2.1 h; the saving comes mostly on easy tasks.

On a 16 GB GPU: APEX-i-quality, effort high and low

The same suites, on an RTX 4060 Ti 16 GB with the experts mostly in system RAM (2× Xeon E5-2670 v2,

125 GB):

| | |

|---|---|

| offload | -ncmoe 42 --load-mode none |

| KV cache | f16 |

| context | 98,304 (the largest request used 56,641) |

| sampling | the card's |

| compaction | off |

| | effort high | effort low |

|---|---|---|

| general (25) | 25/25 | 25/25 |

| hard (5) | 5/5 · 59/59 tests | 5/5 · 59/59 tests |

| frontier (9) | 9/9 · 142/142 tests | 9/9 · 142/142 tests |

| repair/mutation (3) | 3/3 · 27/27 tests | 3/3 · 27/27 tests |

| TOTAL | 42/42 · 253/253 · 347,312 tok · 6.1 h | 42/42 · 253/253 · 235,190 tok · 4.2 h |

Zero failures and zero truncations again. On this box, effort low spent fewer output tokens than

high on 32 of 42 tasks (median ratio 0.80; two-sided sign test p = 0.0009), one run each.

Token totals are not comparable to the GB10 table above: different hardware and different file.

A full score is the entry bar, not the whole result — these suites are built to saturate for a

competent coder. What separates models that all pass is how many tokens they spend, and that

needs repeated runs to compare; the token column above is one run.

Running it

> ### ⚠️ These commands need the FORKED binary — stock llama.cpp cannot load these files

> If you scrolled straight here to copy a command: kolibri1 has no support in upstream

> llama.cpp. Build the fork first:

>

> ```bash

> git clone -b kolibri1 https://github.com/brywil/llama.cpp llama.cpp-kolibri

> cd llama.cpp-kolibri && cmake -B build -DGGML_CUDA=ON && cmake --build build -j

> ```

>

> Details and the verified commit are at the top of this card.

./llama.cpp-kolibri/build/bin/llama-server \

-m Kolibri-1-uniform-Q4_K.gguf \

--ctx-size 131072 --parallel 1 -ngl 999 \

--jinja --reasoning-format deepseek --reasoning on --reasoning-effort high \

--temp 1.0 --top-p 0.97 --top-k 128

• Always pass --ctx-size. The model's metadata declares its 262,144-token native context and

llama.cpp otherwise allocates all of it. At \~20 KiB/token that is only \~5 GiB here, but set it

deliberately.

• It is a reasoning model. The chat template emits <think>…</think>; with

--reasoning-format deepseek the reasoning arrives in reasoning_content and content stays

clean. Thinking can be long, so give it a generous output budget — a response cut off mid-thought

arrives with empty content.

• Tool calling works through llama-server's OpenAI-compatible endpoint with --jinja (the

embedded template); the agentic results above used exactly this path.

• Use the sampling above — it is the original model card's recommendation.

Speed on a DGX Spark (GB10, unified memory), single stream, measured over each full agentic run:

| | uniform-Q4_K | APEX-2.4bpw |

|---|---:|---:|

| generation | \~51 t/s (p10 47.5, p90 53.5) | \~57 t/s |

| prompt processing | \~1,500 t/s overall, \~1,800 on long prompts | — |

On a 16 GB GPU, with the experts in system RAM

Measured on an RTX 4060 Ti 16 GB (PCIe Gen3 x8) with 2× Xeon E5-2670 v2 (AVX, no AVX2) and 125 GB of

RAM, using APEX-i-quality, on a \~5 K-token code prompt (median of 7 runs):

taskset -c 0-9,20-29 ./llama.cpp-kolibri/build/bin/llama-server \

-m Kolibri-1-APEX-i-quality.gguf --parallel 1 \

-ngl 99 --n-cpu-moe 42 -c 131072 -ctk q8_0 -ctv q8_0 -fa on \

-ub 2048 -b 2048 -t 20 -tb 20 --numa isolate --load-mode none \

--jinja --reasoning-format deepseek --reasoning on --reasoning-effort high \

--temp 1.0 --top-p 0.97 --top-k 128

| variant | VRAM | generation | prompt |

|---|---:|---:|---:|

| as above (128 K) | 14.7 GB | 24.7 t/s | 362 t/s |

| --n-cpu-moe 46 -ub 8192 -b 8192 (long fresh prompts) | 13.7 GB | 22.4 t/s | 646 t/s |

| --n-cpu-moe 45 -c 262144 (full native context) | 14.3 GB | 23.4 t/s | 338 t/s |

• Use --load-mode none. This llama.cpp has no --no-mmap, and its default (auto: mmap, plus

lazy reads of large tensors) cost 45% of prompt speed here.

• Generation is CPU-bound. It barely moves across --n-cpu-moe 42–44, so take the value that

leaves \~1.5 GB of VRAM free. f16 KV adds nothing and costs an expert layer, and 10 threads match 20.

• Prompt processing costs \~5–8 s per batch, because each batch streams the CPU-resident experts to

the GPU. Larger -ub means fewer batches.

• taskset / --numa isolate pin to the GPU's NUMA node. Adjust the CPU list for your machine, or

drop it on a single socket.

• uniform-Q4_K is \~2 GB larger. Expect it to need --n-cpu-moe about 2 higher at the same context.

Limitations

1. Agentic results are for uniform-Q4_K, APEX-i-quality and APEX-2.4bpw. The other three files were checked

for coherent German and English generation and a small executed coding test, and measured by

KLD / top-1, but were not run through the agentic suites. Given that the smallest file scored

42/42, the in-between sizes are very likely fine — but that is an inference, not a measurement.

1. The KLD figures rest on 10,240 tokens per corpus. Mean KLD carries roughly ±0.005 at that

size, so the gap between uniform-Q4_K and APEX-i-quality is under 2σ; the gap at \~3.5 bpw is

clearer.

1. The allocation is structural, not measured. The APEX files band experts by depth; no

per-tensor sensitivity map exists for this architecture yet. That is the most likely route to a

better file below 3 bpw.

1. Perplexity is not quoted as a quality figure. BF16 scores 27.96 on raw WikiText-2 — normal

for a reasoning chat model reading unformatted text, and not comparable to other models' numbers.

The KLD and top-1 columns, measured against this model's own BF16, are the meaningful ones.

Provenance

• Converted from the original BF16 safetensors with the fork's converter. The port was validated

layer by layer against an independent fp32 PyTorch implementation written from Aleph Alpha's own

vLLM model code: identical expert selection in every layer sampled through layer 16, full-layer

outputs agreeing to \~1e-3 at every layer, and the same top-9 next-token ranking.

• Quantized with quantkit (dynamic_apex.py recipes, the

memory-bounded banded imatrix generator serialized_gen_kolibri.py) and the fork's

llama-quantize. The imatrix was cross-checked against stock llama-imatrix (dense tensors

correlate 0.9996–1.0000, routed experts median 0.995–0.999).

• KLD reference logits from the BF16 GGUF; WikiText-2 test split, 1,290,590 bytes,

sha256 173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08.

• Licensed Apache-2.0, as the original. All credit for the model to Aleph Alpha.