---

license: apache-2.0

base_model:

- jzinno/Ornith-1.5-35B-A3B-DFlash2

- ornith-ai/Ornith-1.5-35B-A3B

tags:

- llama.cpp

- gguf

- speculative-decoding

- dflash

- dflash2

- ornith

---

Ornith-1.5-35B-A3B DFlash2 draft model — GGUF for llama.cpp

A llama.cpp-compatible GGUF of jzinno's DFlash2 speculative-decoding drafter for Ornith-1.5-35B-A3B — standalone conversion with the target's tokenizer cloned in and a reproducible conversion script.

Converted from the original safetensors checkpoint (jzinno/Ornith-1.5-35B-A3B-DFlash2, vLLM-only format) to the dflash GGUF architecture.

> Priority note (corrected Sept 14): an earlier version of this card claimed to be the first llama.cpp GGUF of this drafter — wrong: gbuzhf's APEX repo has shipped Q8_0/BF16 conversions since Aug 26, 2026. This repo (published Sept 13) is an independent conversion; credit where due.

• Drafter: 0.5B dense, 6 layers, block_size 16, 8 target layers, mask token [PAD248077]
• License: Apache-2.0 (same as source)

Usage

Requires a llama.cpp build with DFlash2 support (PR #27816, b10896+ / v0.4.0):

llama-server -m Ornith-1.5-35B-A3B-IQ3_XXS-vcruz.gguf \
  --spec-type draft-dflash \
  --spec-draft-model Ornith-1.5-35B-A3B-DFlash2-draft-jzinno-F16.gguf \
  --spec-draft-n-max 4 \
  --cache-type-k q4_0 --cache-type-v q4_0 -ub 512

The cache/ubatch flags are the 16 GB recipe: they free ~1 GB so the target + drafter + KV fit in VRAM at large ctx. On 24 GB+ cards you can omit them. Pick one speculative path (DFlash2), never combine with draft-mtp.

A Q8_0 variant of the drafter (~0.5 GB smaller) is available in gbuzhf's APEX repo — we run it in production on v0.4.0 (loads and drafts normally in live traffic). On 16 GB cards the freed ~0.5 GB measurably delays the VRAM-overflow slowdown at large in-flight contexts.

Benchmarks (RTX 5080 16 GB, greedy, IQ3_XXS target, n_max=4)

Measured 2026-09-13, single-variable runs, every row confirmed by at least 2 independent runs:

| Allocated ctx | Default flags (KV q8_0, ub 2048) | Lean recipe (KV q4_0 + ub 512) |

|---|---|---|

| 2K | 318 tok/s ✓ fits | — |

| 16K | 316 tok/s ✓ fits | — |

| 32K | 50 tok/s ⚠️ VRAM overflow → WDDM paging | 328 tok/s ✓ fits |

| 50K | 53 tok/s ⚠️ VRAM overflow → WDDM paging | 326 tok/s ✓ fits |

The drafter wins (+70–75% over raw) at EVERY context size — the only failure mode is VRAM overflow. With default flags, target (~13.5 GB) + drafter (~1.1 GB) + KV + compute buffers cross the 16 GB line around ctx ≈ 20K, and Windows WDDM silently pages instead of crashing (draft rounds then cost ~61 ms instead of ~12.5 ms — the ⚠️ cells; the allocated size matters, not the token count in KV). Free ~1 GB with --cache-type-k q4_0 --cache-type-v q4_0 -ub 512 (already in the Usage command above) and the same 16 GB card runs 326 tok/s at 50K ctx. On 24 GB+ cards the default flags should sustain large ctx directly.

Accepted/step (llama-server mean len): 3.1–4.0 tokens per verification round; acceptance 53–74% (74% on short code-gen, ~53% on long-form), independent of the recipe.

Practical rule: watch your memory budget, not your context size. If target + drafter + KV + buffers fit, use the drafter; if they overflow, slim the recipe (KV q4_0, smaller -ub) or stay raw.

> Corrections (2026-09-13, same day). v1 of this card reported 160/135/57 tok/s from ad-hoc runs with uncontrolled flags — superseded. v2 reported the speeds above but attributed the large-ctx drop to a "kernel regime cliff" and advised keeping ctx ≤ 16384 — wrong: it was plain VRAM overflow (silent WDDM paging), fully removed by the lean recipe. This version also restructures the table by recipe so it can't be skimmed as "useless above 16K" — the drafter is useful at every ctx it fits in VRAM.

Who benefits

• Anyone whose target + drafter + KV fits in VRAM: +70–75% on a 5080 at any ctx (up to 50K verified with the lean recipe). Acceptance (53–74%) doesn't depend on GPU speed, so the ratio should hold or improve on slower cards.
• The only failure mode is memory: if the package overflows, WDDM/swap makes drafting counterproductive — slim the recipe (KV q4_0, smaller -ub) or stay raw.

Reports welcome — open a discussion with your GPU, --ctx-size, KV recipe and numbers.

For reference, the source checkpoint reports accepted lengths up to 4.87 (n=7) on H200/vLLM against both BF16 and NVFP4 targets.

Conversion notes (reproducibility)

• Weights written as-is from pytorch layout (gguf-py derives ne from the reversed numpy shape — do not transpose)
• tokenizer.ggml.mask_token_id = 248077 (from dflash_config.mask_token_id in the source config.json)
• Norm and *_conv_base tensors stored in F32 (binary-op operands in the dflash graph require matching F32 types; F16 triggers a binbcast assert on CUDA / "unsupported types f32/f16" on CPU)
• Conversion script: convert_from_safetensors.py in this repo