---

pipeline_tag: text-generation

library_name: gguf

base_model: z-lab/Qwen3.5-397B-A17B-DFlash

base_model_relation: finetune

license: apache-2.0

inference: false

tags:

- dflash

- speculative-decoding

- speculative-decoding-draft

- draft-model

- block-diffusion

- gguf

- ik_llama.cpp

- qwen3.5

- ornith

- iq2_ks

- moe

---

DFlash Drafter for Ornith 1.0 397B A17B IQ2_KS (GGUF)

**This repository contains a drafter, not a standalone language model. It does not produce

useful output by itself. Load the recommended mixed precision file

Ornith-DFLASH-m1-q4.gguf with Ornith 1.0 397B A17B IQ2_KS through --model-draft.**

Initialized from z-lab/Qwen3.5-397B-A17B-DFlash and retrained by distillation against an

IQ2_KS target on an NVIDIA GB10.

At n_max=3 the recommended file decodes at **25.65 tokens per second against 18.71 with

speculation disabled, a 37.0% increase**. Against the upstream drafter at matched engine

configuration, the retrained weights measured +6.1% across four boots, with a prompt level

95% interval of +1.7% to +10.5% that is conditional on those boots rather than a general

estimate. Both arms were FP16 so that only the weights differ. Details and raw

data for both figures are below.

Usage

llama-server \
  --model    Ornith-1.0-397B-A17B-IQ2_KS-00001-of-00005.gguf \
  --model-draft Ornith-DFLASH-m1-q4.gguf \
  --spec-type dflash:n_max=3,cross_ctx=512 \
  --n-gpu-layers 99 --n-gpu-layers-draft 99 \
  --draft-params "-wgt 17" -wgt 8 \
  --flash-attn on --parallel 1

All five target shards must sit in the same directory. --model names shard one and the

remaining four are loaded automatically by filename.

The two wgt flags reserve worst case graph space for the target and draft contexts. They

were present in the measurements reported below, so omitting them will not reproduce those

numbers. bench/serve-bench.sh carries the full configuration.

--parallel 1 is a hard runtime restriction, not a performance preference. This

ik_llama.cpp build rejects multiple speculative slots at startup, because the

speculative accept path indexes the batch absolutely. It does not merely serialize them.

Compatibility

Retrained against the IQ2_KS build's own output distribution rather than the bf16 model, so

it is tuned to that specific quantized target. Only IQ2_KS results are reported here.

The two quantization axes are distinct. The target is IQ2_KS. The drafter files here

are FP16, Q8_0, and a mixed type described below.

Files

| File | Size | Notes |

|---|---:|---|

| Ornith-DFLASH-m1-q4.gguf | 737 MB | recommended default |

| Ornith-DFLASH-m1-q8_0.gguf | 1.38 GB | benchmark variant |

| Ornith-DFLASH-m1-f16.gguf | 2.59 GB | FP16 conversion, source for the two above, and arm B of the A/B below |

The file named q4 is not a standard Q4 type. It contains no Q4_* tensors at all.

Verified tensor composition: 26 F32, 24 Q8_0, 12 IQ3_KS, 7 IQ4_KSS. The IQ3_KS and

IQ4_KSS types are specific to ik_llama.cpp and will not load in mainline llama.cpp.

Checksums

7b9f2654dfc07c0a9197da3e6414db58202a60008a24f76387a52136f302134a  bench/Ornith-DFLASH-rt-f16-rope1e7.gguf
c2d623da10ee1008a311ccd3806c4501cbe02eec9075fb44cce4819dc13c1934  Ornith-DFLASH-m1-q4.gguf
8b6bee4dc82ca1c89a782212144495e3b874465995dad8e781562d75f452f5de  Ornith-DFLASH-m1-q8_0.gguf
186b73dce2046b072a67361ddb8f09992761eed0171f2cfbb1d251fb8f943d6e  Ornith-DFLASH-m1-f16.gguf

---

Measurement 1: what the retraining bought

A preregistered A/B. The endpoint, design and analysis were fixed in writing before any of

this data existed, so the reported effect is not a best-of-grid selection. It is published as

bench/PREREGISTRATION-2026-08-09-retrain-ab.md. Note the chronology is self attested: no

third party timestamp was obtained before the run.

Arms. Both FP16, with byte-identical engine relevant metadata

(rope.freq_base = 1e7, attention.sliding_window = 4096,

sliding_window_pattern = TTTTTF). They differ in weights only.

| arm | weights | file | sha256 (first 16) |

|---|---|---|---|

| A | upstream z-lab | bench/Ornith-DFLASH-rt-f16-rope1e7.gguf, published here | 7b9f2654dfc07c0a |

| B | retrained | Ornith-DFLASH-m1-f16.gguf | 186b73dce2046b07 |

Design. Counterbalanced replicated boots in the order A, B, B, A. Two boots per arm, so

between-boot variance is measured rather than assumed to be zero. The drafter is a startup

time flag, so a within-boot A/B is impossible. 8 prompts, 2 repetitions per cell per boot,

greedy, 256 predicted tokens, config order reversed on even repetitions.

Primary endpoint, prespecified: paired by prompt difference at n_max=3. That value was

already the established operating point for this artifact before any A/B data was collected,

so it was not selected from these results. For transparency it does turn out to be arm A's

highest observed mean here, while arm B peaks at n_max=4.

| n_max | A, upstream | B, retrained | paired difference | sd | 95% CI | role |

|---:|---:|---:|---:|---:|---|---|

| 0 | 18.03 | 18.06 | +0.039 | 0.072 | [-0.021, 0.099] | secondary |

| 2 | 20.75 | 21.46 | +0.714 | 0.892 | [-0.032, 1.460] | secondary |

| 3 | 21.63 | 22.94 | +1.315 | 1.142 | [0.360, 2.270] | primary |

| 4 | 21.26 | 22.95 | +1.686 | 1.549 | [0.391, 2.981] | secondary |

**The primary endpoint is +1.315 tokens per second, about +6.1%, with the confidence

interval excluding zero.** In relative terms it spans roughly +1.7% to +10.5%.

That interval is computed across the 8 prompts after averaging each arm's two boots, so it

is conditional on these four boots and does not propagate boot-to-boot uncertainty. With

only two boots per arm that component cannot be estimated properly; the observed spread is

reported below so a reader can judge it directly.

Two checks support it:

• The n_max=0 row is compatible with zero, +0.039 with an interval spanning zero. That

cell disables speculation and exercises no drafter, so a large difference there would have

indicated a problem with the arms. It is a sanity check, not evidence of absence, and is

deliberately not used as a correction factor for anything on the speculative path.

• Between-boot spread is small next to the effect. At n_max=3 the two boots of arm A

differed by -0.093 and the two boots of arm B by +0.403, against a +1.315 signal.

n_max=2 is reported as not significant. n_max=4 and n_max=0 were secondary and

exploratory. The larger n_max=4 difference is not promoted to the headline.

Raw rows: bench/results-retrain-ab-20260809.csv, 256 rows, each carrying run_id,

boot_index, arm, drafter filename, drafter sha and server binary sha.

Measurement 2: what the shipped artifact delivers

Absolute throughput of Ornith-DFLASH-m1-q4.gguf, the recommended file, measured

2026-08-09 in a single server boot with n_max varied per request. 16 runs per cell,

8 prompts times 2 repetitions, ± is the sample standard deviation.

| n_max | 0 | 1 | 2 | 3 | 4 | 6 | 8 | 12 | 15 |

|---|---|---|---|---|---|---|---|---|---|

| tok/s | 18.71 | 21.26 | 24.62 | 25.65 | 25.23 | 24.24 | 21.04 | 19.83 | 18.72 |

| ± | 0.23 | 0.46 | 1.23 | 1.61 | 2.40 | 2.63 | 3.49 | 3.79 | 3.50 |

| output tokens per round | n/a | 1.87 | 2.56 | 3.06 | 3.39 | 3.96 | 4.42 | 4.82 | 4.89 |

The last row is total output tokens per verification round, computed as

predicted_n / (predicted_n - accepted). It includes the token the target produces itself, so

accepted draft tokens per round is exactly one lower.

Speculation on against speculation off, paired by prompt within this single boot, is

+6.93 tokens per second, 95% CI [5.55, 8.32], a 37.0% increase, positive on all 8

prompts with a per prompt range of +20.8% to +52.5%. Because both conditions come from one

server lifetime with one binary and one artifact, this comparison carries none of the

cross-session caveats attached to Measurement 1.

The shipped mixed precision file measured faster than the FP16 file used in the A/B, 25.65

against 22.94 at n_max=3, but those come from different experiments and different boots and

no controlled quantization comparison was run, so this is an observation rather than an

attribution. FP16 was used for the A/B only to isolate weights. **This A/B does not cover the

shipped quantization.**

Why n_max=3

What the published rows establish directly is the shape: step time grows close to linearly in

n_max while accepted tokens per round grows sublinearly, so throughput peaks and then falls.

The usual explanation, which this data does not itself prove, is that decode here is memory

bound and the target is a sparse mixture of experts, so each verify position routes to a

different subset of experts and speculation cannot amortize expert weight traffic the way it

amortizes the dense remainder. Treat that mechanism as a hypothesis consistent with the curve

rather than a result of it.

Fitting step time to the published rows in bench/results-m1q4-20260809.csv, where step

time is predicted_ms / (predicted_n - accepted), gives approximately

90.9 + 14.7 x (n_max - 1) ms over K=1 to K=12. The linear form is only a local

approximation. Measured minus fit, in ms: -2.9 at K=1, -1.7 at K=2, -1.0 at K=3, -0.7 at

K=4, -1.1 at K=6, +16.3 at K=8, -9.9 at K=12 and -36.6 at K=15. It tracks the interior

closely and breaks down at the ends, so it should not be extrapolated.

n_max=3 is the highest observed mean for this artifact on this GB10 configuration, and is

the recommended setting. It is not established as a strict optimum: its paired lead over

n_max=4 is only +0.414 tokens per second with a confidence interval of [-0.568, 1.395].

---

Reproducing

Published in bench/.

| | |

|---|---|

| serve-bench.sh | standalone launcher carrying every measured flag literally |

| bench.py, prompts.json | request driver and the 8 prompt coding set |

| run_sweep.sh | used only for Measurement 2; it boots through an unpublished launcher |

| PREREGISTRATION-2026-08-09-retrain-ab.md | the preregistration for Measurement 1 |

| results-retrain-ab-20260809.csv | Measurement 1 raw rows, 256 |

| results-m1q4-20260809.csv | Measurement 2 raw rows, 144 |

| engine-local-patches.diff | the local engine changes, 3301 lines |

Prompts cover Python, Rust, SQL, JavaScript, and code review. Sampling is

temperature = 0, seed = 42, max_tokens = 256. n_max is varied per request through

"speculative.n_max": K against a server booted with a high startup N_MAX, which acts as

the per request ceiling on this recurrent model.

Engine. ik_llama.cpp at base commit

f96eaddba8bed6a9a5e628bbf6a566775c70b49c plus local patches across 21 files,

+1967 / −279 lines, published as bench/engine-local-patches.diff. The base commit alone

does not reproduce the measured binary. Built 2026-08-05 with -DGGML_CUDA=ON,

CMAKE_CUDA_ARCHITECTURES=native which resolves to sm_121 on GB10, and

-march=armv8.2-a+dotprod+i8mm+fp16. Binaries are dynamically linked, so all four objects

matter:

7b7ff16cd57264905bf3b37334a29856189eb004f0cbb0b3c3cc4d04c951a576  llama-server
6ab80960540e043ebd8001162ab3401cb267a61a53b513ce13ebb1cd2fc1c089  libllama.so
1e7e686140afdd4069956414238a8e674d04136492967ba1b2cfde635cdcebca  libggml.so
3f160b1f9dc7df27142da9a0d0e1345a7cf143557f526d77a75c1e79c5e3b305  libmtmd.so

The drafter_sha16 and server_sha16 columns in the CSVs carry 16 character prefixes of

these values, not full digests; the full digests are here and in the Checksums block above.

Target. paragon-of-brah/Ornith-1.0-397B-DFLASH-GGUF,

the five Ornith-1.0-397B-A17B-IQ2_KS-0000N-of-00005.gguf shards, sizes 19982543552,

19806331648, 19806331648, 19994372480 and 18176416800 bytes respectively. That repository is

mutable and the Hub revision was not recorded at measurement time, so the sizes above are

the only pin offered; treat the target as identified but not cryptographically fixed.

Which launcher produced which measurement. Measurement 1, the A/B, was run through the

published serve-bench.sh directly, so its launcher and full configuration are published.

That is not the same as full reproducibility: the target repository is mutable and its

revision was not recorded, so the target is pinned only by shard sizes. Measurement 2,

the absolute curve, was run through run_sweep.sh, which boots via an unpublished launcher; serve-bench.sh reproduces the configuration that script resolved, but that

equivalence has not been verified by re-measuring Measurement 2 through it.

Every result row carries run and configuration identifiers. Note that bench.py appends to

its output file and does not fail fast on a short grid, so always point it at a fresh path.

---

Training

The method below is readable in train/train_dflash.py, which is published here, so the

objective, schedule and hyperparameters can be checked against code. What cannot be checked

are the inputs and the original run: the prompts, feature pack and training logs are not

published. Counts and step timings from that run are therefore omitted rather than stated as

unverifiable figures, and the throughput measurements above are the claims this card stands

on.

• Method: distillation from the IQ2_KS target. Target features were captured from an

IQ2_KS server and the DFlash head was retrained to predict the target's own argmax

continuations. The target model is frozen throughout, as are the shared target embedding

and output head, which are not drafter parameters. Teacher and student are different

networks: the teacher is the 397B target, the student is the 6 layer draft head.

• What the feature pack contains: target hidden state vectors and target token IDs. It

does not contain drafter activations; those are produced online during training.

• Objective: masked block cross entropy with position decay weights

w_k = exp(-(k-1)/gamma) and block size 16, which weights early draft positions most

heavily.

• Prompts: short coding tasks written for this project, covering Python, Rust, SQL,

JavaScript and code review. The set is not published, so treat any characterization of its

contents as author reported.

• Anchors: the trainer samples only anchors possessing a complete context ring within a

single capture segment, and each anchor additionally needs the block width in rows after it

to supply targets, so anchor yield per captured row is low.

• Optimizer: AdamW with warmup then cosine decay, gradient clipping, and trainable

parameters held in fp32 under bf16 autocast. Note the schedule multiplies warmup by a

cosine that decays from step zero rather than applying cosine after warmup, so the nominal

peak learning rate is never actually reached. The exact schedule is in

train/train_dflash.py and can be read there.

• Initialization: upstream z-lab bf16 weights. Saved weights are bf16, excluding rotary

and inv_freq buffers.

Checkpoint selection

The best checkpoint was chosen by an offline acceptance metric computed on a fixed anchor

subset taken from a capture separate from the training pack. Because that same subset both

guided selection and produced the resulting score, the score is selection validation rather

than an independent estimate, and neither it nor the per anchor data behind it is published

here. The selection procedure is readable in train/train_dflash.py; the throughput

measurements above are the claims this card

stands on.

---

Conversion requirement

The Hugging Face config for a DFlash drafter carries rope_theta nested inside

rope_parameters. convert_hf_to_gguf.py reads the top level key. A straight conversion

therefore writes the wrong rope frequency, and the model loads, runs, and quietly drafts

badly with no error message.

The config used for these builds carries rope_theta: 10000000.0 at the top level in

addition to the nested rope_parameters block. If you convert your own drafter and

acceptance looks poor, check this first. Note this config was prepared separately for

conversion; the trainer's in memory adjustment of rope_theta does not reach any file,

because checkpoint saving writes only weights and the best checkpoint record.

---

Provenance and license

Apache-2.0, inherited from the upstream drafter

z-lab/Qwen3.5-397B-A17B-DFlash,

a joint Z-Lab, Modal and SGLang release. See the paper

and code. The target model is

deepreinforce-ai/Ornith-1.0-397B,

itself derived from Qwen/Qwen3.5-397B-A17B.

This repository contains a derivative of the upstream drafter weights, retrained as

described above.

Reproducibility of training. The trainer is published in train/ as

train_dflash.py with its featpack.py reader, verified against the compiled artifact of

the original run. It is unmodified, so it still contains a local sys.path insert that you will need to

adjust for your environment.

Two dependencies are not included. dflash.py, which provides DFlashDraftModel, is

upstream z-lab code and should be taken from

z-lab/dflash; the copy used here has sha256

80d58268f8839a22cad42516cc95d127b79fca194d25ee7a8cc4d610bc8f560e. The feature packs and

the target embedding and output head dumps are omitted because of their size, several

gigabytes each, so the training run cannot be reproduced from this repository alone.

Not affiliated with Z-Lab, Modal, SGLang, Qwen, or deepreinforce-ai.