---

license: mit

base_model: inclusionAI/Ling-3.0-flash

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- llama.cpp

- ling

- bailingmoe3

- mixture-of-experts

- linear-attention

- kda

- mla

- reasoning

quantized_by: prometheusAIR

---

Ling-3.0-flash-GGUF — sm_120-safe quants

Quants of inclusionAI/Ling-3.0-flash,

a 127.5B-total / 5.1B-active hybrid MoE (BailingMoeV3, llama.cpp arch

bailingmoe3).

Every file here contains only sm_120-safe tensor types — no iq1_s,

iq2_s or iq3_s, which are broken on NVIDIA Blackwell (RTX PRO 6000,

RTX 5090, RTX 5070 Ti). All were quantized from the BF16 master, never from

another quant, and declare a verified 256K context window.

> ### ⚠️ Re-download if you pulled these before 2026-08-17

> These files were republished on 2026-08-17 and the previous ones are

> obsolete in two independent ways:

>

> 1. They will not load. Upstream llama.cpp merged official support

> (#26608) as arch

> bailingmoe3. The old files declared bailing-hybrid, a name we invented

> before support existed, and used tensor names (ssm_f, ssm_g) that did

> not survive review. No patch is needed any more — build stock llama.cpp.

> 2. They were missing the SwiGLU clamp — a real quality bug, not a rename.

> See The SwiGLU clamp below.

>

> Nothing about the weights themselves changed: the fix was metadata, so the

> tensor data in these files is byte-for-byte what it was. You still need the

> new files, because the old metadata cannot be loaded or corrected in place.

Which one do I want?

| | Q4_K_M | Q5_K_M | Q5_K_XL | Q5_K_XXL | Q8_0 |

|---|---|---|---|---|---|

| Weights | 69.70 GiB | 84.18 GiB | 88.92 GiB | 92.78 GiB | 126.32 GiB |

| + KV at full 256K | +1.97 GiB | +1.97 GiB | +1.97 GiB | +1.97 GiB | +1.97 GiB |

| Mean KLD vs BF16 | 0.015854 | 0.007082 | 0.005726 | 0.005069 | 0.002966 |

| 99th pct KLD | 0.159557 | 0.071339 | 0.058221 | 0.052040 | 0.032671 |

| Top-1 agreement | 95.09 % | 96.74 % | 97.07 % | 97.35 % | 97.98 % |

| PPL ratio vs BF16 | 1.0128 | 1.0043 | 1.0038 | 1.0021 | 0.9988 |

Reference PPL 4.1516 ± 0.0461, from the BF16 master with the clamp active.

• Q5_K_XXL — the most accurate build that fits a 96 GB card at the full 256K

window. Recommended for a dedicated RTX Pro 6000; it leaves ~1.4 GiB spare.

• Q5_K_M — the balanced pick, and the right one if anything else shares the

GPU (~10.5 GiB spare).

• Q5_K_XL — in between, ~5.9 GiB spare.
• Q4_K_M — smallest. The Q4→Q5 step is about twice the Q5→Q8 step on both

mean KLD and top-1, so if you are deciding where to spend memory, that is the

one that matters.

• Q8_0 — reference fidelity. Needs expert offload or two GPUs.

KV cache is unusually cheap here — only 7 of 42 layers are MLA, and those are

compressed — so the full 256K window costs under 2 GiB. Weights are what

limit you on this model, not context.

The SwiGLU clamp

config.json carries expert_swiglu_limit_list and

share_expert_swiglu_limit_list, which clamp the SwiGLU activations on the

deepest layers: routed experts at 4.0 on layers 35-41, shared experts at 5.0 on

34-39 and 7.0 on 40-41. The op is

silu(gate).clamp(max=L) * up.clamp(-L, L).

**Files published before 2026-08-17 did not carry these values, so the clamp

never fired.** The mistake is worth describing, because the same trap is

waiting in any port of this family: inclusionAI's own

modeling_bailing_moe_v3.py does not implement it — BailingMoeV3MLP.forward

is a plain SwiGLU with zero references to the limit lists — so reading the

reference implementation is not enough to catch it. vLLM's bailing_moe_v3.py

implements it (SwigluStepAndMul), and so does the merged llama.cpp. Two

independent runtimes against one incomplete reference file.

Measured on Q5_K_XXL, clamped against unclamped, on identical weights:

| | unclamped (old files) | clamped (these files) |

|---|---:|---:|

| Mean KLD | 0.009187 | 0.067203 |

| Median KLD | 0.001318 | 0.001453 |

| 99.9% KLD | 0.676011 | 11.027311 |

| PPL | 4.404 | 4.159 |

The effect is rare but severe: the median barely moves while the 99.9th

percentile goes up by 16×, which is exactly what a clamp on outlier activations

should look like. Perplexity drops 5.4%.

At full precision the same thing shows up independently — the BF16 master scores

PPL 4.1516 with the clamp against 4.3970 without it, on identical

weights. A 5.6% perplexity improvement is not what a spurious operation does.

There is a second-order benefit: **clamping also makes the model easier to

quantize.** Measured against a matched reference, every tier's divergence

roughly halves versus the old unclamped-vs-unclamped numbers — Q5_K_XXL goes

0.00919 → 0.00507 mean KLD, Q8_0 0.00668 → 0.00297 — because bounding the

activations removes the outlier paths where quantization error was amplified.

Running it

Point llama.cpp at shard 00001; it finds the rest. Omitting -c gives the

full 262,144-token window.

./build/bin/llama-server -m Ling-3.0-flash-Q5_K_XXL-00001-of-00003.gguf -ngl 99

Reasoning is on by default — allow enough completion tokens or you will get

truncated or empty content. Sampling per the upstream model card:

temperature 0.6, top_p 0.95, top_k 20.

If -ngl 99 does not fit your card you have two dials, and **context is the

cheaper one to keep**:

• Offload experts to CPU — -ncmoe N keeps the experts of N blocks in

system RAM. Experts are 97% of the weights, so this is the big lever. Costs

speed, not context.

• Reduce context — saves comparatively little, since the whole 256K window

is under 2 GiB of KV. Reach for this last.

Rough sizing, weights + KV at full 256K: Q4_K_M ≈ 72 GiB, Q5_K_M ≈ 86 GiB,

Q5_K_XL ≈ 91 GiB, Q5_K_XXL ≈ 95 GiB, Q8_0 ≈ 128 GiB, plus a couple of GiB of

compute buffers.

Config for a single 96 GB card (RTX PRO 6000 Blackwell)

Specific to a 96 GB card with nothing else resident

./build/bin/llama-server \
  -m Ling-3.0-flash-Q5_K_XXL-00001-of-00003.gguf \
  -ngl 99 -c 262144 -ub 512 --no-mmap

Measured on these exact files, upstream b10470: 95,696 MiB resident at the

full 256K window with no CPU offload, 125 tok/s decode. Prefill was

~2,040 tok/s on a 235K prompt when last measured, before the migration.

-ub 512 is load-bearing at this size. -ub is the compute-buffer dial and its

price scales with context: at 262,144 each ubatch token costs ~0.695 MiB, since

the attention mask is ctx × ubatch and is materialised even with -fa on

(measured: 256→512 = +178 MiB, 512→1024 = +356 MiB). At 32K it is roughly a

tenth as expensive. Lowering -ub costs prefill throughput and leaves decode

untouched. When tuning this close to the limit, confirm with an actual

generation rather than a successful load — CUDA allocates some workspace lazily

on the first matmul.

Context: 256K, not 131K

These GGUFs give you 262,144 tokens by default. You do not need any flag.

This used to require a patch on our side: config.json originally shipped

max_position_embeddings: 131072, contradicting inclusionAI's own card, which

documents a training schedule of 8K → 32K → 256K and benchmarks at a 256K

window. inclusionAI has since corrected the config to 262144, so these files

get the right window straight from the converter.

It matters because llama.cpp caps a slot to the GGUF's context_length:

the slot context (262144) exceeds the training context of the model (131072) - capping

Verified by needle-in-a-haystack out to 247,583 tokens.

To use the old 131072 value instead:

--override-kv bailingmoe3.context_length=int:131072

Files

Each quant is split into shards for resumable download.

| file | size |

|---|---:|

| Ling-3.0-flash-Q4_K_M-*-of-00002.gguf | 69.70 GiB |

| Ling-3.0-flash-Q5_K_M-*-of-00003.gguf | 84.18 GiB |

| Ling-3.0-flash-Q5_K_XL-*-of-00003.gguf | 88.92 GiB |

| Ling-3.0-flash-Q5_K_XXL-*-of-00003.gguf | 92.78 GiB |

| Ling-3.0-flash-Q8_0-*-of-00004.gguf | 126.32 GiB |

The bailing-hybrid-llama.cpp.patch that used to live here has been removed.

It was written against 6ea215d17, before upstream support existed, and no

longer applies to master. Build stock llama.cpp instead.

How these were made

Experts are 97.12% of all parameters, so the recipe spends its budget there

and protects everything else cheaply: all 3.67B non-expert parameters are pinned

to q8_0 for only 3.6 GiB. That covers the KDA gates (ssm_f_a/ssm_g_a),

the MLA absorption tensors (attn_k_b/attn_v_b), the head-wise attention gate,

embeddings and output. Norms, the router (ffn_gate_inp), exp_probs_b and the

KDA scalars (A_log, dt_bias, conv1d) stay F32.

Pinning attn_k_b also avoids a silent fallback: its ncols is 128, not

divisible by 256, so K-quants degrade it to q5_1/q4_1.

The expert projections are where the tiers differ:

| | down_exps | gate_exps | up_exps |

|---|---|---|---|

| Q5_K_M | q5_K | q5_K | q5_K |

| Q5_K_XL | q6_K | q5_K | q5_K |

| Q5_K_XXL | q6_K | q5_K | q6_K on layers 9–41 |

down_proj is promoted first because it is the output side of the SwiGLU, where

error propagates instead of being gated away. up_proj follows: in

down(silu(gate(x)) ⊙ up(x)) an error in up passes through the product with

its relative magnitude intact, while an error in gate is first attenuated by

silu.

Q5_K_XXL stops where it does because of a hardware ceiling. Promoting up on

all 40 MoE layers costs another 892 MiB, which forces -ub 256 and drops prefill

by roughly a quarter for no measurable accuracy gain. Full Q6_K experts needs

~101.4 GiB resident against 97,887 MiB of card, and since the entire 256K KV

cache is only ~2 GiB, no amount of context reduction rescues it. The seven layers

left at q5_K are the seven lowest by imatrix activation energy — Σ(Act²) of

11K–24K against 193K at layer 40.

imatrix from 892K tokens of mixed code, agent traces and technical prose. With

top-8 routing over 512 experts, 40 of 120 expert tensors reached only

98.24–99.80% coverage — 1–9 experts per tensor never routed to, concentrated in

the deepest layers. Those tensors (blk.36/40/41) are quantized one step higher

so missing importance data is paid for in bits, costing ~1.1 GiB.

The imatrix was collected before the clamp was known about, so it was re-checked

rather than assumed: collecting a matched pair with and without the clamp moves

per-column importance only in the extreme head (Spearman ≥ 0.9992, top-1%

column overlap 97–98%, and only 4 of 573 tensors shifting more than 1% in total

activation energy — all of them ffn_down_exps on layers 38–41). Bit allocation

lands in the same place, so these quants were not rebuilt.

Validation

From a single RTX PRO 6000 Blackwell 96 GB. Q4/Q5 run all-GPU; Q8_0 with

-ncmoe 16.

• Needle-in-haystack 12/12 at depths 10/50/90% across 6.6K, 28K, 61K and

123.4K tokens, and 12/12 again at 175K and 247.6K — which is what justifies

the 262,144 context_length these files declare. Measured on Q4_K_M, Q5_K_M

and Q8_0; Q5_K_XXL passes at 235,379 tokens and Q5_K_XL at 179,167.

• Coherence 6/6 — factual recall, arithmetic, the bat-and-ball trap, code

generation, exact-word instruction following, translation.

Divergence figures in the table above come from

llama-perplexity --kl-divergence against reference logits from the **BF16

master itself**, not a Q8_0 proxy — quant-vs-quant comparisons overstate damage

because the errors are correlated. 160 chunks × 512 tokens of held-out

wikitext-2 (~82K tokens, zero verbatim overlap with the imatrix corpus).

The reference was regenerated on 2026-08-17 with the clamp active; figures

published before that date were measured against an unclamped reference and do

not carry over.

That ordering is solid and monotonic in bit-width. A 20-instance multi-hop

reasoning probe at ~111K tokens could not separate the Q5 tiers from one

another, so treat KL divergence as the evidence for quality ordering, and do not

read the gaps between adjacent Q5 tiers as a promise of visibly better answers.

Not measured: tool calling, and reasoning quality between 128K and 256K —

retrieval is verified to 247.6K, but reasoning was only probed at ~111K.

MTP: works, predicts well, and is still slower — leave it off

The layer-42 nextn head is present in every file here. Upstream bailingmoe3

can actually use it (--spec-type draft-mtp), which our own port never could,

so it is now measurable. On Q5_K_M at 32K, single stream:

| --spec-draft-n-max | decode | draft acceptance |

|---|---:|---:|

| off | 135.9 tok/s | — |

| 1 | 108.3 tok/s | 95.8 % |

| 2 | 111.4 tok/s | 92.5 % |

| 4 | 109.2 tok/s | 88.1 % |

| 6 | 109.6 tok/s | 88.1 % |

The head drafts well — 96% accepted at depth 1 — and it is a ~20% net loss

anyway, at every depth. The flat curve is the tell: if the cost scaled with

drafted tokens, depth 1 would sit near baseline. It does not, so this is a fixed

per-step cost of running the MTP graph at all — an extra graph launch, a

512-expert gather, and a projection over a 157K vocab, paid once per decode step

regardless of how many tokens it drafts.

Worth stating plainly because it is the opposite of the usual intuition:

acceptance rate is not the metric. Measure tokens per second.

Enabling MTP also costs ~2 GiB of VRAM, since the layer-42 tensors go from

"unused tensor — ignoring" to actually allocated. That alone rules it out

alongside Q5_K_XXL on a 96 GB card, which has ~1.4 GiB spare.

Architecture notes

42 layers: 35 KDA (Kimi Delta Attention) + 7 gated MLA at layers

5/11/17/23/29/35/41, plus an MTP/nextn head at layer 42. 512 routed experts,

top-8 plus 1 shared, noaux_tc grouped routing (8 groups, 4 used), sigmoid

scoring.

Three things differ from Kimi-Linear and matter to anyone porting this:

• kda_safe_gate replaces -exp(A_log)*softplus(·) with

lower_bound sigmoid(exp(A_log) (f(x) + dt_bias)), lower_bound = -5.0.

A_log is stored as +exp(A_log); the sign lives in

bailingmoe3.kda.gate_lower_bound.

• rope_interleave: true resolves to llama.cpp's NORM rope, not the

NEOX that DeepSeek-style MLA usually uses.

• attention.g_proj exists on both layer types with different shapes —

[4096,2560] KDA full-rank output gate vs [32,2560] MLA head-wise attention

gate. The converter sends them to different tensors (ssm_g_a and

attn_gate); a name→enum table cannot express that.

Building llama.cpp

Any master build from 2026-08-17 onward has bailingmoe3 support. No patch.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

License

MIT, following the base model.