---

base_model: inclusionAI/Ling-3.0-flash

base_model_relation: quantized

license: mit

library_name: gguf

pipeline_tag: text-generation

quantized_by: aj9o9

tags:

- gguf

- moe

- ling

- bailing

- imatrix

---

Ling-3.0-flash - GGUF (Q4_K_M, IQ3_XXS)

GGUF quantization of inclusionAI/Ling-3.0-flash,

a 124B parameter Mixture-of-Experts model with 5.1B active parameters per token.

A note from me

This is my first quantization, and my first time patching llama.cpp. I did it to learn, and I

learned a lot - including that I got part of it wrong, found it, and fixed it (see the warning

below). I am sharing it because a working GGUF of this model did not exist when I started, not

because I think it is authoritative.

Please report anything you find. Bad output, crashes, wrong metadata, better flags for a

particular card - open a discussion on this repo or reach me at

@ItsmeAjayKV. Suggestions are as welcome as bug reports.

On the llama.cpp side: I am not planning to submit this fork upstream. The architecture support

was written with heavy AI assistance and the model is very new, so

PR #26608 is the right place for upstream

support to come from - it is independent work by someone else, it has had real review, and it

handles the KDA gate parameters more cleanly than my branch does. My branch exists so these files

have something to run on, and so I could understand how the pieces fit together.

> [!IMPORTANT]

> This will not load in upstream llama.cpp. Ling 3.0 uses a hybrid KDA + MLA attention

> architecture that upstream does not yet support. These files declare the architecture

> bailing-hybrid, which currently exists only in the fork linked below. Loading them with

> a standard llama.cpp build fails with unknown model architecture.

> [!WARNING]

> If you cloned the fork before 2026-08-08, pull again.

> Earlier builds had the KDA forget gate wrong. Ling 3.0 sets kda_safe_gate: true, which

> selects lower_bound sigmoid(exp(A_log) (g + bias)) - lower_bound scales the sigmoid,

> it is not a clamp floor. The original build used the other branch of the reference kernel,

> -exp(A_log) * softplus(g + bias) plus a hard clamp, across 35 of the 42 layers.

>

> Both versions produce values in the same range, so output stayed fluent and the problem was

> not visible from casual use. The curves differ substantially though: on a neutral token the

> old gate retained roughly 50% of the recurrent state where the reference retains roughly 8%.

>

> Fixed in b94e0ea. **The GGUF files

> here are unaffected** - the bug was in the graph, not the weights, so there is nothing to

> re-download.

> [!NOTE]

> The fix builds and runs, but it has not been validated numerically against the reference

> implementation. A proper check means comparing logits against HF or vLLM on the same tokens,

> which I have not been able to run. If you can do that comparison, I would genuinely like to

> see the result.

Files

| Quant | Files | Total | Bits/weight | imatrix |

|---|---|---|---|---|

| IQ3_XXS | Ling-3.0-flash-IQ3_XXS.gguf (single file) | 47.9 GB | 3.08 | yes |

| IQ4_XS | Ling-3.0-flash-IQ4_XS-0000{1,2}-of-00002.gguf | 66.4 GB | 4.27 | yes |

| Q4_K_M | Ling-3.0-flash-Q4_K_M-0000{1,2}-of-00002.gguf | 75.2 GB | 4.84 | no |

For split quants, download every shard into the same directory and point -m at the first

one - llama.cpp finds the rest automatically. Do not rename them. IQ3_XXS is a single file.

IQ4_XS is smaller than Q4_K_M and imatrix-guided, so it is the better pick of the two if it

fits your memory. Q4_K_M is kept because it was made first and some people prefer K-quants.

Note on IQ3_XXS: 7 of 917 tensors (blk.{5,11,17,23,29,35,41}.attn_k_b) fell back to iq4_nl

because their 128 columns are not divisible by the 256-value block size IQ3_XXS requires. Those

are the MLA layers, and they end up with slightly more precision than requested.

Which one? If you have 64 GB of RAM, take IQ3_XXS - it fits, and a quant that fits in RAM

beats a better quant served from disk by a wide margin. Above that, Q4_K_M is the better model.

Also here:

| File | Size | What it is |

|---|---|---|

| ling.imatrix | 466 MB | The importance matrix used for IQ3_XXS. Reusable - see below. |

How these were made

Both quants come from the same bf16 GGUF, converted from the original safetensors with the

converter in the fork linked below.

Q4_K_M was quantized directly, with no importance matrix.

IQ3_XXS requires one (llama-quantize refuses to build it without), so:

bf16 (249 GB) --quantize--> Q8_0 (123 GiB) --llama-imatrix--> ling.imatrix
                                                                   |
bf16 (249 GB) -----------------quantize with imatrix---------------+--> IQ3_XXS

The Q8_0 exists only because the bf16 does not fit in the memory available for an inference pass.

Q8_0 is near-lossless, so the activations it produces stand in for bf16's. The final quantization

reads the bf16, not the Q8_0 - stacking two lossy steps would waste quality that a 3-bit quant

cannot spare.

Calibration: calibration_data_v5_rc.txt

(Dampf, building on kalomaze), 230 chunks at -c 512. This corpus was chosen deliberately: it

mixes 20+ languages, code, config and mathematics, and v5 exists specifically to improve expert

activation in sparse MoE models.

That matters here. Ling routes 8 of 512 experts per token, so a narrow corpus leaves most

experts with no measurements at all, and those get quantized blind. Measured coverage:

| Calibration | Expert coverage |

|---|---|

| 3 chunks | 62-67% |

| ~100 chunks | ~85% |

| 230 chunks (final) | 99.4-99.8% |

Reusing the imatrix: ling.imatrix is the expensive artifact here - about 25 minutes on an

H100. If you want a different size (IQ4_XS, IQ2_M, Q3_K_M), download it and pass --imatrix

rather than recalibrating. It is architecture-specific to Ling-3.0-flash but not quant-specific.

Usage

Build the fork:

git clone -b ling-3.0-bailing-hybrid https://github.com/Ajay9o9/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-completion

Run it. These files are 48-75 GB, so unless you have that much VRAM, offload the experts to CPU

with --n-cpu-moe - only 8 of 512 experts are read per token, so this costs less than it sounds:

./build/bin/llama-completion \
  -m Ling-3.0-flash-IQ3_XXS.gguf \
  -p "Write a haiku about linear attention." \
  -ngl 99 --n-cpu-moe 42 -n 100

Lower the --n-cpu-moe number to keep more experts on the GPU if you have VRAM spare.

On a 96 GB card with Q4_K_M fully resident, this model runs at roughly 113 tok/s.

Hardware notes

I made this quant with consumer cards in mind - specifically a 3060 (12 GB) and a 3090 (24 GB),

paired with 64 GB of system RAM. Because only 8 of 512 experts run per token, pushing the expert

weights to CPU costs far less throughput than the size suggests.

| Setup | Expect |

|---|---|

| 3090 (24 GB) + 64 GB RAM | Workable. Use --n-cpu-moe 42, lower it if VRAM allows. |

| 3060 (12 GB) + 64 GB RAM | Tight but runs. Keep --n-cpu-moe 42 and expect a slow prompt phase. |

| 64 GB RAM, no GPU | Possible via mmap, but slow. |

This is why IQ3_XXS exists. Q4_K_M is 75.2 GB, which does not fit in 64 GB of RAM, so part of

it is served from disk through mmap on every token. IQ3_XXS fits with room to spare. On a machine

with 64 GB that difference matters more than the difference in quantization quality - an SSD helps,

a spinning disk will crawl, and neither beats simply fitting in memory.

Rule of thumb: pick the largest quant that fits in RAM, not the largest quant you can download.

About the model

• 42 layers: 35 KDA (linear attention, recurrent state) interleaved 5:1 with 7 gated MLA layers
• 512 routed experts + 1 shared, 8 experts active per token, group-limited noaux_tc routing
• 256K context. Only the 7 MLA layers keep a KV cache, and they store a 576-value compressed

latent per token rather than full per-head keys and values, so the cache stays small at long

context.

See the original model card for benchmarks and

training details. Those numbers were measured on the full bf16 model with the reference

implementation and do not describe this quantization.

Links

• Original model: inclusionAI/Ling-3.0-flash
• Fork required to run this: Ajay9o9/llama.cpp @ ling-3.0-bailing-hybrid
• Upstream request: ggml-org/llama.cpp#26590
• Alternative implementation: ggml-org/llama.cpp#26608
• Reference implementation: inclusionAI/vllm-ling-v3

License

MIT, inherited from the original model.