---

license: apache-2.0

base_model: z-lab/Qwen3.6-35B-A3B-DFlash

library_name: llama.cpp

tags:

• dflash
• dflash2
• speculative-decoding
• draft-model
• ornith-1.5-35b-a3b

---

Ornith-1.5-DFlash2-Draft-GGUF

DFlash 2 draft model trained on the ornith-1.5-35b-a3b family, step119 checkpoint.

Model details

• Base: z-lab/Qwen3.6-35B-A3B-DFlash
• Architecture: DFlash2DraftModel
• Block size: 16
• Hidden size: 2048
• Num attention heads: 32
• Num key value heads: 8 (GQA)
• Head dim: 128
• Intermediate size: 6144
• Sliding window: 4096
• Vocab size: 248320
• Draft vocab size: 248320
• Rope theta: 10000000
• Dtype: bfloat16

DFlash config:

• conv_group_size: 16
• conv_kernel_size: 2
• mask_token_id: 248077
• selector_rank: 256
• selector_top_k: 16
• target_layer_ids: [1, 6, 11, 16, 22, 27, 32, 37]
• block_size: 16

Provided files

| File | Size | SHA256 |

|---|---|---|

| ornith-dflash2-bf16.gguf | 1,063,708,128 | 2845841c37a7bd37c275e726ebb2cbea13c52156568ec20e56c58229f1625dfc |

| ornith-dflash2-f16.gguf | 1,063,708,128 | ad45b593b6f0f88c0785ac2c15b9e114ec9f11ddd84e2d0020bd06999d8c73cd |

| ornith-dflash2-q4_0.gguf | 307,406,304 | 0b3bd6726ac1f321df5f3b780c4645a0602c2dbfbf2147f221cb6b10baa47aec |

| ornith-dflash2-q8_0.gguf | 570,467,808 | 26d3264c2a5887be8d55c3810fd2a792f01a111fe08cb31974735db3957c855e |

Benchmarks

Measured 2026-10-07 on a single RTX 4090 24 GB, q4_0 against q4_0. Summary first, then the

full protocol, then the per-probe detail and the limits of what these numbers support.

Result at 160K context, draft length 4

Target model: Ornith-1.5-35B-Q4_K_M. Decoding greedy (temperature 0, top_k 1). Five

agentic-coding probes of ~150,000 tokens each, 3 reps each.

| Draft | Quant | Acceptance | Accepted/draft | Decode tok/s (median) |

|---|---|---|---|---|

| ornith-dflash2 (this repo, step119) | q4_0 | 0.471 | 1.88 | 113.8 |

| z-lab/Qwen3.6-35B-A3B-DFlash (v1) | q4_0 | 0.457 | 1.82 | 112.1 |

Prefill was ~2.2k tok/s in both arms (2,210 and 2,205), and is independent of the draft.

The two drafts are equivalent on this workload, not separated. The pooled acceptance delta

is +0.0148 (+3.2% relative), but that is carried by two probes: pairing probe-by-probe, this

checkpoint is ahead on only 2 of 5, and the median paired delta is -0.0017. The honest

reading of these rows is "as good as the v1 draft at this draft length", not "better".

How it was benchmarked

| | |

|---|---|

| Target model | Ornith-1.5-35B-Q4_K_M (35B-A3B MoE, Q4_K_M GGUF) |

| Draft models | this repo's ornith-dflash2-q4_0.gguf vs z-lab/Qwen3.6-35B-A3B-DFlash quantized to q4_0 with the same tool |

| Hardware | one RTX 4090, 24 GB |

| Runtime | llama.cpp build 11541, commit c094ce598, branch master |

| Context | 160,000 tokens, one full prompt per request (cache_prompt off, so every request re-prefills) |

| Sampling | greedy (temperature 0, top_k 1; the seed is irrelevant because it is deterministic) |

| Prompt corpus | 5 agentic-coding probes, ~150,000 tokens each, assembled from real repository source (call-graph tracing, bottleneck ranking, reproduction construction, a design note) and truncated to an exact token budget with the target model's own tokenizer |

| Draft length | --spec-draft-n-min 4 --spec-draft-n-max 4 (fixed, to isolate the draft model) |

| KV cache | --cache-type-k q4_0 --cache-type-v q4_0, rope/state cache f16, draft KV q8_0 |

| Other flags | --n-gpu-layers 88, --ubatch-size 256, --n-cpu-moe 0, --spec-draft-ngl 999, --spec-draft-backend-sampling |

| Reps | 3 generations per probe -> 15 per arm, 30 total |

| Generation length | up to 512 tokens per request; all 30 generations ran to exactly 512 |

| Arms | identical in every flag except --spec-draft-model |

Only --spec-draft-model moves between the two arms, so the unit of analysis is the per-probe

delta and nothing else has to be controlled for.

Per-probe detail

Acceptance per probe. The head-to-head sign flips, which is why the pooled figure is

misleading on its own:

| Probe | step119 | v1 | Delta |

|---|---|---|---|

| 1 | 0.474 | 0.506 | -0.0315 |

| 2 | 0.419 | 0.421 | -0.0017 |

| 3 | 0.678 | 0.627 | +0.0505 |

| 4 | 0.401 | 0.403 | -0.0023 |

| 5 | 0.444 | 0.378 | +0.0654 |

Mean paired delta +0.0161, median -0.0017. Deltas are given to 4 dp, the same precision as the

acceptance figures they come from, because at 3 dp several of them land exactly on a rounding

boundary.

Acceptance is deterministic under greedy decoding. The per-probe ratios repeat exactly

across all three reps, so the 3 reps measure throughput jitter only (~1-2% run to run); they add

no statistical power to acceptance. The real dispersion is probe to probe: 0.40-0.68 for both

drafts. Quoting a single acceptance figure without that range overstates the precision.

Serving two concurrent 160K sessions

The same q4_0 draft, the same context, two streams at once — the shape an agentic harness actually

runs. Target Ornith-1.5-35B-Q4_K_M, prompts xl160-0 and xl160-1 (150 001 tokens each, verified

with /tokenize rather than character counts), draft length 4, one round per arm, peak VRAM and wall

time for the whole round.

| Server config | streams | verdict | peak VRAM | wall | end-to-end tok/s |

|---|---|---|---|---|---|

| --ctx-size 184800 --parallel 2 --kv-unified (the shape most setups ship) | 1 | ok | 23 830 MiB | 72.7 s | 7.04 |

| same | 2 | rejected — 500 Context size has been exceeded | — | — | — |

| --kv-unified-per-slot 160000 --parallel 2, --n-cpu-moe 4, --ubatch-size 128 | 2 | ok | 23 472 MiB | 512.1 s | 2.00 |

| --kv-unified-per-slot 160000 --parallel 2, --n-cpu-moe 3, --ubatch-size 128 | 2 | ok | 23 962 MiB | 441.5 s | 2.32 |

| --kv-unified-per-slot 160000 --parallel 2, --n-cpu-moe 4, --ubatch-size 256 | 2 | ok | 23 988 MiB | 325.4 s | 3.15 |

| --kv-unified-per-slot 160000 --parallel 2, --n-cpu-moe 3, --ubatch-size 256 | 2 | fails to load (cudaMalloc) | — | — | — |

Why row 2 fails: --kv-unified means one shared KV pool, so --parallel 2 does not give you two of

anything. 184 800 cells cannot hold two 150 001-token prompts plus their generations; the server

admits both slots and then rejects both requests with Context size has been exceeded. To serve

two 160K sessions, size the pool per slot: --kv-unified-per-slot 160000 --parallel 2 (pool =

320 000), and omit --ctx-size — the flag only auto-sizes the pool when the context is unset.

What that costs, on a 24 GB card: the larger pool needs a larger KV cache and larger compute

buffers, which only fit with MoE layers offloaded to CPU, and the offload forces a smaller

--ubatch-size. Prefill is what the ubatch scales (~1 047 tok/s at ub 256 versus ~632 at ub 128,

after offload), and a 150K prompt is almost entirely prefill — so end-to-end throughput lands

below the single-stream baseline (3.15 vs 7.04 tok/s). The best configuration found puts

4 of 40 MoE layers on the CPU at --ubatch-size 256, not the 3 layers at ub 128 that a

fit-only search would pick: fewer offloaded layers forces a smaller ubatch, and the prefill penalty

outweighs the layer saved. Measure the wall, not the offload count.

These are capacity figures. Two 160K sessions become possible at all — the shipped shape cannot

serve them at any speed — but neither is fast, and the pair together is slower than one baseline

session. Do not read this as a throughput improvement.

Measurement notes:

• Rates are end-to-end — generated tokens divided by the whole round's wall time, prefill

included. At 150K the prefill is the wall, so a decode-phase rate is not comparable.

• Do not compare per-stream server rates under concurrency. timings.predicted_per_second for

two identical requests read 2.2 and 43.7 tok/s, because llama-server charges the slot admitted

first for the other slot's prefill. The wall-clock aggregate is the honest number.

• Speculative acceptance also falls with a second stream: 2.168 -> 1.723 accepted tokens per

draft step (1 -> 2 streams), since the draft's batch spans both slots.

• Peak VRAM is flat across every successful arm (23 472-23 988 MiB, ~98.5% of the card), so the

configuration changes the shape of the cost rather than the total. The winning config leaves

388 MiB of headroom.

• Reproduce with `llama-server --kv-unified-per-slot 160000 --parallel 2 --kv-unified

--cache-type-k q4_0 --cache-type-v q4_0 -ctr f16 -cts f16 --ubatch-size 256 --n-gpu-layers 98

--n-cpu-moe 4 --spec-type draft-dflash --spec-draft-model <this repo, q4_0> --spec-draft-ngl 999

--spec-draft-n-min 4 --spec-draft-n-max 4 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0

--spec-draft-backend-sampling`.

Limitations

• No speed-up ratio is published. A no-draft reference was not measured at 160K, so no

speed-up vs no-speculation figure is claimed here. A short-context no-draft number is not a

valid 160K denominator and is not divided into these results.

• The draft length was held at 4, and the Usage snippet below recommends 2. Both drafts

accept only ~1.85 of 4 drafted tokens, so most of the n=4 tail is wasted for both, which is a

plausible reason they measure equal. The two draft lengths have not been compared head to

head, and a shorter draft length is where a difference would most likely show.

• One workload class (agentic coding at ~150K tokens), one quant pair, one machine, five probes.

These numbers are not a general claim about either draft model.

• Throughput figures are for this context length on this GPU; they will not transfer to other

context lengths or accelerators.

Reproduce

Both arms at --ctx-size 160000 and --spec-draft-n-min 4 --spec-draft-n-max 4:

llama-server \
  --model Ornith-1.5-35B-Q4_K_M.gguf \
  --ctx-size 160000 \
  --n-gpu-layers 88 \
  --cache-type-k q4_0 --cache-type-v q4_0 -ctr f16 -cts f16 \
  --ubatch-size 256 --n-cpu-moe 0 \
  --spec-type draft-dflash \
  --spec-draft-model ornith-dflash2-q4_0.gguf --spec-draft-ngl 999 \
  --spec-draft-n-min 4 --spec-draft-n-max 4 \
  --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 \
  --spec-draft-backend-sampling

A build without DFlash draft support will reject --spec-type draft-dflash; the results above

were produced with build 11541 (commit c094ce598).

Usage

Use with llama.cpp speculative decoding:

llama-server --model /path/to/base.gguf --ctx-size 204800 --n-gpu-layers 88 --spec-type draft-dflash --spec-draft-model /path/to/ornith-dflash2-q4_0.gguf --spec-draft-ngl 999 --spec-draft-n-max 2 --spec-draft-n-min 1 --spec-draft-backend-sampling

For the measured configuration — including the flags needed to serve **two concurrent 160K

sessions** on a single 24 GB card, and why --n-cpu-moe and --ubatch-size must be chosen

together — see Serving two concurrent 160K sessions above.

License

Apache-2.0, matching the DFlash family.