---

license: mit

license_link: https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B/blob/main/LICENSE

base_model: deepreinforce-ai/Ornith-1.0-35B

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- llama-cpp

- quantized

- text-generation

- coding

- mtp

- qwen3_5_moe

- single-gpu

- tp1

---

Ornith-1.0-35B — GGUF (llama.cpp, single-GPU tp=1)

Single-GPU llama.cpp GGUF package for deepreinforce-ai/Ornith-1.0-35B.

The supported serving policy is tp=1 — one model copy per GPU. Multi-GPU

tensor-parallel serving is intentionally out of scope for this version.

This release ships six body quants (Q3_K_M → Q8_0) plus an **integrated

IQ4_XS-MTP graft** that adds a native multi-token-prediction (MTP) draft head for

low-concurrency speculative decode. Quant quality is measured against the upstream

BF16 GGUF with a native llama.cpp next-token top-64 KL-divergence probe over 32

coding prompts.

TL;DR — pick an artifact

| Use case | Recommended artifact | Key numbers |

|---|---|---|

| Default serving speed | ornith-1.0-35b-Q4_K_M.gguf | 19.71 GiB on disk, 21.31 GiB loaded VRAM, 243.3 tok/s c1, 655.6 tok/s c16 |

| Lowest memory | ornith-1.0-35b-Q3_K_M.gguf | 15.61 GiB on disk, 17.27 GiB loaded VRAM, 240.5 tok/s c1, 493.0 tok/s c16 |

| Middle footprint | ornith-1.0-35b-IQ4_XS.gguf | 17.64 GiB on disk, 19.34 GiB loaded VRAM, 0.1426 mean top-64 KLD nats |

| Highest fidelity / footprint | ornith-1.0-35b-Q6_K.gguf | 26.56 GiB on disk, 28.03 GiB loaded VRAM, 0.0165 mean top-64 KLD nats, 32/32 top-1 |

| Native low-concurrency MTP | ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf | 19.6 GB decimal, 0.0731 mean top-64 KLD (29/32), c1/128 adaptive MTP 319.53 tok/s, acceptance (0.953, 0.865) |

> MTP graft note: active MTP is byte-for-byte identical to target-only at the

> next-token (API-visible) level, and gives ~1.3× single-stream decode. It is

> not bit-exact to target-only over long deterministic sequences — see

> MTP. Use target-only for strict reproduction,

> MTP for throughput on open-ended work.

Quantization & fidelity

Fidelity is KL(P_bf16 || P_candidate) over next-token top-64 distributions,

32 coding prompts, n_predict=1, temperature=-1, n_probs=64, keyed by token ID.

Lower KLD and higher greedy top-1 agreement are better.

Primary fidelity table

| Artifact | Mean KLD (nats) | Greedy top-1 † |

|---|---:|---:|

| Q8_0 | 0.0108 | 31/32 (96.9%) |

| Q6_K | 0.0165 | 32/32 (100.0%) |

| Q5_K_M | 0.0354 | 30/32 (93.8%) |

| IQ4_XS-MTP (graft) | 0.0731382442 | 29/32 (90.6%) |

| Q4_K_M | 0.0864 | 29/32 (90.6%) |

| IQ4_XS | 0.1426 | 27/32 (84.4%) |

| Q3_K_M | 0.3620 | 27/32 (84.4%) |

!Next-token fidelity ladder: mean top-64 KLD vs BF16 for the six body quants plus the IQ4_XS-MTP graft, lower is better.

*Fidelity ladder — the integrated IQ4_XS-MTP graft (purple) lands between Q5_K_M

and Q4_K_M, i.e. better next-token fidelity than its Q4_K_M neighbor.*

> † Top-1 vs mean KLD are different axes. Top-1 counts how often the single

> most-likely (argmax) token matches BF16's, over the 32 prompts; mean KLD

> measures divergence across the full top-64 distribution. Two quants can share a

> top-1 count yet differ a lot in KLD — e.g. Q3_K_M and IQ4_XS are both

> 27/32 on top-1, but Q3_K_M's distribution diverges ~2.5× more (0.3620 vs

> 0.1426 nats). Neither value is a typo.

• The **integrated IQ4_XS-MTP graft is 48.70% lower mean KLD than the plain

IQ4_XS body** (0.0731382442 vs 0.1425748206), and improves greedy top-1 from

27/32 to 29/32.

• Active draft-mtp ≡ target-only at the next-token level: their top-64

next-token distributions are identical (KLD 0.0, 32/32) for the one-token

API-visible probe.

Body-quant distribution detail (secondary)

Full per-quant distribution from the corrected top-64 probe (same 32 prompts).

Mean / P50 / P95 / Max are explicit nats statistics of the per-prompt top-64

KLD; in this run the BF16 top-64 distribution captured mean probability mass

0.999965, so the top-64 approximation captured essentially all next-token mass.

| Quant | GGUF GiB | KLD-probe VRAM GiB | Mean nats | P50 nats | P95 nats | Max nats | Top-1 † |

|---|---:|---:|---:|---:|---:|---:|---:|

| Q3_K_M | 15.61 | 16.32 | 0.3620 | 0.2548 | 1.1077 | 1.3730 | 27/32 (84.4%) |

| IQ4_XS | 17.64 | 19.34 | 0.1426 | 0.0868 | 0.3195 | 0.6586 | 27/32 (84.4%) |

| Q4_K_M | 19.71 | 20.35 | 0.0864 | 0.0379 | 0.2877 | 0.4503 | 29/32 (90.6%) |

| Q5_K_M | 23.03 | 23.61 | 0.0354 | 0.0235 | 0.0943 | 0.2497 | 30/32 (93.8%) |

| Q6_K | 26.56 | 27.07 | 0.0165 | 0.0092 | 0.0513 | 0.0586 | 32/32 (100.0%) |

| Q8_0 | 34.37 | 34.77 | 0.0108 | 0.0052 | 0.0440 | 0.0590 | 31/32 (96.9%) |

The IQ4_XS-MTP graft uses the same probe on a different breakpoint context; its

distribution detail (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511) is

in the MTP section. Source:

benchmarks/kld-quant-vs-bf16-top64.md.

Serving performance

Aggregate decode throughput and p95 time-to-first-token across concurrency,

llama.cpp tp=1, from scripts/bench_openai.py --stream --max-tokens 256 (0

failed requests per row). Q4_K_M is the speed pick; Q5_K_M nearly ties it at c16.

| Quant | c1 tok/s | c1 p95 TTFT ms | c4 tok/s | c4 p95 TTFT ms | c8 tok/s | c8 p95 TTFT ms | c16 tok/s | c16 p95 TTFT ms |

|---|---:|---:|---:|---:|---:|---:|---:|---:|

| Q3_K_M | 240.5 | 77.9 | 422.0 | 170.7 | 464.4 | 344.9 | 493.0 | 493.7 |

| IQ4_XS | 234.1 | 75.1 | 297.7 | 159.7 | 411.5 | 330.2 | 476.0 | 541.5 |

| Q4_K_M | 243.3 | 76.3 | 458.3 | 192.3 | 615.0 | 361.8 | 655.6 | 650.0 |

| Q5_K_M | 236.7 | 75.1 | 311.0 | 198.8 | 439.0 | 383.6 | 638.6 | 620.4 |

| Q6_K | 225.9 | 76.8 | 295.8 | 194.2 | 409.6 | 394.8 | 603.3 | 657.4 |

| Q8_0 | 208.5 | 76.9 | 281.5 | 190.8 | 405.8 | 389.1 | 601.4 | 725.8 |

!Aggregate decode throughput vs concurrency (c1–c16) for the six quants; Q4_K_M leads at c16.

!p95 time-to-first-token vs concurrency (c1–c16) for the six quants; lower is better.

> Profile note: the table above is a short-context profile — `CTX_SIZE=8192

> PARALLEL=16 exposes n_ctx = 512`/slot, with ~23-token prompts and 256-token

> generations. Source: benchmarks/llamacpp-quant-benchmarks.md.

> For how prefill latency scales with prompt length, see long-context TTFT below.

Long-context TTFT (single stream)

p95 time-to-first-token vs prompt/context length, single GPU tp=1, single stream

(CTX_SIZE=131072 PARALLEL=1, n_ctx/slot=131072, exact prompt token counts, 140

rows/quant, 0 failures).

| Context tokens | Q4_K_M p50/p95 ms | Q4_K_M tok/s | IQ4_XS p50/p95 ms | IQ4_XS tok/s | MTP-graft p50/p95 ms | MTP-graft tok/s |

|---:|---:|---:|---:|---:|---:|---:|

| 512 | 91.8 / 94.1 | 188.3 | 87.1 / 88.7 | 177.4 | 86.0 / 87.8 | 184.0 |

| 1024 | 169.6 / 172.2 | 145.8 | 159.9 / 161.6 | 139.9 | 157.6 / 159.0 | 155.9 |

| 2048 | 341.8 / 346.1 | 97.9 | 316.1 / 318.7 | 112.0 | 307.6 / 308.9 | 113.9 |

| 4096 | 699.2 / 702.2 | 67.2 | 647.0 / 653.4 | 70.7 | 625.7 / 631.3 | 69.0 |

| 8192 | 1447.5 / 1458.0 | 33.7 | 1342.9 / 1348.5 | 39.8 | 1297.4 / 1304.0 | 40.9 |

| 16384 | 3009.7 / 3030.4 | 19.5 | 2793.2 / 2804.0 | 20.9 | 2709.8 / 2737.3 | 21.5 |

| 32768 | 6302.4 / 6313.0 | 9.7 | 5829.9 / 5853.1 | 10.5 | 5673.5 / 5696.6 | 10.7 |

!Long-context p95 TTFT vs prompt length (512–32768 tokens) for Q4_K_M, IQ4_XS, and the MTP graft; single stream, tp=1.

p95 TTFT rises with prompt length (prefill cost) to ~6.3 s at 32k tokens for

Q4_K_M; the IQ4_XS body and IQ4_XS-MTP graft prefill slightly faster at every

length. Decode throughput falls from ~180–190 tok/s at 512 tokens to ~10 tok/s at

32k as the KV cache grows. Source: benchmarks/llamacpp-longctx-ttft.md.

Multi-token prediction (MTP)

The integrated IQ4_XS-MTP-graft-headQ6 artifact carries a native MTP draft head

on the IQ4_XS body. The release recommendation is adaptive: use MTP for

low-concurrency / single-user requests and keep LLAMA_SPEC_MAX_DRAFTING_SLOTS=1

so saturated batches fall back to target-only throughput. The MTP catalog

validates the published MTP profiles.

Next-token quality

Active MTP does not change the one-token, API-visible next-token

distribution. Mean BF16 KLD is 0.0731382442 nats over the 32-prompt top-64

probe (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511), top-1 29/32. The

active draft-mtp vs target-only graft next-token KLD is 0.0 (32/32 identical).

Sequence-level behavior

A deterministic 8 prompt × 64 token probe (`n_predict=64, temperature=-1,

n_probs=64) compares the same GGUF running active native draft-mtp` against

target-only.

| Runtime | Client agg tok/s | Server decode tok/s | Draft acceptance | Exact 64-tok seqs | Token-position match |

|---|---:|---|---|---:|---|

| Target-only graft | 172.57 | ~210 (per-request timing) | n/a | 8/8 baseline | n/a |

| Fast active draft-mtp, n_max=2 | 233.81 | 325.70 (/metrics) | 310/378 = 82.01%; per-position (0.884, 0.747) | 6/8 | 478/512 = 93.36% |

| LLAMA_SPEC_VERIFY_SEQUENTIAL=1 | 164.34 | 204.88 (/metrics) | 313/375 = 83.47%; per-position (0.884, 0.772) | 5/8 | 455/512 = 88.87% |

!MTP single-stream decode speedup on the deterministic 8x64 probe: target-only vs active MTP graft, client and server decode.

• Fast active MTP gives ~1.35× client throughput (233.81 vs 172.57) and

server decode 325.70 vs ~210, with 82.01% draft acceptance.

• It matches 6/8 sequences exactly and 478/512 = 93.36% of token

positions; the two non-exact sequences first diverge at positions 60 and

34.

• The LLAMA_SPEC_VERIFY_SEQUENTIAL=1 variant is both slower (164.34 /

204.88) and matches less (5/8, 88.87%; divergences at 25, 43, 60) — i.e. the

fast verifier is here both faster and more target-matching.

> Note: MTP self-speculation is **not bit-exact to target-only over long

> deterministic generations**. For workloads that require exact target

> reproduction, run target-only; for throughput on open-ended work, MTP wins.

> MTP is a single-user / low-concurrency win — it is not faster on saturated

> batches, which is why the adaptive throttle falls back to target-only.

Sequence logprob comparability (API limitation)

llama.cpp omits candidate top_logprobs for most accepted speculative tokens, so

a naive "sequence KLD" looks enormous — that is an **API/logprob-availability

artifact, not model quality**. On the comparable subset where the emitted token

also matches, KLD is ≈ 0.001 nats (near zero):

| Runtime | Paired positions | Comparable top-logprobs | Missing | Comparable same-token mean KLD (nats) | Max (nats) |

|---|---:|---:|---:|---:|---:|

| Fast active draft-mtp vs target-only | 512 | 12 | 500 | 0.0011215739 | 0.0040239103 |

| Sequential verifier vs target-only | 512 | 10 | 502 | 0.0012293172 | 0.0040239103 |

Full sequence KLD cannot be computed without changing the runtime to return

target verifier logprobs for accepted speculative tokens.

Draft-head distillation provenance

The meaningful KL-style metric for a standalone MTP draft head is **teacher KL

against cached target hidden states**, not standard base-model next-token KLD.

The draft head was trained and measurably improved across checkpoints —

teacher KL fell 1.92 → 1.50 → 0.76 and teacher argmax top-1 rose 0.54 → 0.69 →

0.89.

| Draft checkpoint | Tokens | Windows | Teacher KL | Teacher argmax top-1 | Teacher argmax top-5 | Gold top-1 |

|---|---:|---:|---:|---:|---:|---:|

| mtp-distill-kl-step500 | 22,637 | 128 | 1.9222025748 | 0.5360250917 | 0.7881786456 | 0.4425939833 |

| snapshots-iq4-live-accepted-a/step-1000 | 58,754 | 413 | 1.5038724942 | 0.6876808388 | 0.8368451510 | 0.6764816013 |

| snapshots-iq4-live-allrows-b/step-1000 | 32,264 | 2,048 | 0.7591610373 | 0.8908070915 | 0.9389722291 | 0.8076183982 |

!MTP draft-head distillation progress: teacher KL falls (left) while top-1 agreement rises (right) across three checkpoints.

*Teacher KL is measured against cached target hidden states (not base next-token

KLD). These training artifacts live under artifacts/mtp/ locally and are

referenced by name as provenance — they are not committed in this HF repo.*

Run with llama.cpp

Serve a body quant on GPU0 (tp=1):

QUANT=Q4_K_M PORT=8000 CTX_SIZE=8192 PARALLEL=16 REASONING=off \
  scripts/serve_llamacpp_gpu0.sh

Serve the integrated MTP graft for low-concurrency speculative decode (keep

adaptive throttling so saturated batches fall back to target-only):

QUANT=IQ4_XS-MTP-graft-headQ6 PORT=8000 CTX_SIZE=8192 PARALLEL=1 \
  CACHE_RAM=0 REASONING=off LLAMA_SPEC_MAX_DRAFTING_SLOTS=1 \
  scripts/serve_llamacpp_gpu0.sh

REASONING=off is the default because the model otherwise spends simple coding

prompts in reasoning_content before producing final content. See

benchmarks/llamacpp-q4-reasoning-off-fix.md.

Serving / quant / MTP profile catalogs:

configs/serving_profiles.yaml,

configs/quant_artifacts.yaml,

configs/mtp_profiles.yaml.

Benchmark environment

Hardware: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB

VRAM, single GPU (tp=1). Driver 580.159.03, CUDA runtime reported by driver 13.0.

Dedicated MTP KLD eval config (2026-06-28): local llama.cpp CUDA server,

single RTX PRO 6000 Blackwell 96GB, tp=1, CTX_SIZE=8192, PARALLEL=1,

CACHE_RAM=0, REASONING=off.

• llama.cpp build/commit: 050ee92d04c2e1f639025786dea701c70e7d4204 (pinned for

the long-context TTFT run; earlier KLD/throughput sweeps were not separately pinned).

• Reasoning mode: all benchmarks in this card ran with REASONING=off.

Provenance & reproducibility

• Figures: regenerate with scripts/make_charts.py

→ assets/02_fidelity_ladder.png, assets/03_throughput_tps.png,

assets/04_ttft_p95.png, assets/05_mtp_draft_head_distill.png,

assets/06_mtp_tps.png.

• Serving throughput / TTFT: benchmarks/llamacpp-quant-benchmarks.md.
• Body-quant fidelity: benchmarks/kld-quant-vs-bf16-top64.md.
• Dedicated MTP KLD / sequence eval (2026-06-28):

benchmarks/mtp-kld-eval-2026-06-28.md

and benchmarks/mtp-dedicated-kld-evaluation.md,

with raw evidence under benchmarks/raw/. Includes the

sequence-level probe and its logprob-availability limitation (full sequence KLD

is not computable from the current llama.cpp API response).

• Adaptive MTP serving profile: benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md.
• Long-context TTFT: benchmarks/llamacpp-longctx-ttft.md.