---

license: mit

base_model: inclusionAI/Ling-3.0-tiny

base_model_relation: quantized

tags:

- rocmfpx

- ai-max-395

- ryzen-ai-max-395

- amd

- gguf

- llama.cpp

- rocm

- gfx1151

- strix-halo

- bailingmoe3

- moe

- hybrid-linear-attention

language:

- en

pipeline_tag: text-generation

---

> ### ⚠️ STOCK llama.cpp WILL NOT LOAD THIS MODEL

> bailingmoe3 is not merged upstream, and Ling-3.0-tiny additionally needs the

> Q-LoRA attention path (q_lora_rank: 256) that flash-era builds lack. Ignore the

> auto-generated "Use this model" commands above — use the patch in patches/.

>

> 🚀 101.08 tok/s on AMD Ryzen AI MAX+ 395 (gfx1151 / Strix Halo) —

> 4.30 GiB, smaller than Q4_K_M (4.49 GiB). Verified on two independent machines.

✅ The patch you need is in this repo

patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch — applies to

charlie12345/ROCmFPX at commit 2809dc5,

verified with git apply --check.

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout 2809dc5
git apply patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch
cmake -B build -S . -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

Full build notes, per-architecture details and licence: patches/README.md in this repo.

⚠️ If you add files under src/models/, re-run cmake -B build -S . — the models/*.cpp GLOB

is configure-time, so cmake --build alone will not link them.

---

Ling-3.0-tiny — ROCmFP4 (tier 102 COHERENT) GGUF

A 4-bit ROCmFP4 quantization of inclusionAI/Ling-3.0-tiny,

built for AMD gfx1151 (Ryzen AI MAX+ 395 / Strix Halo) with per-tensor protection of the

LM head, token embeddings, shared experts, router, and the hybrid model's recurrent state.

| | |

|---|---|

| File | Ling-3.0-tiny-Q4_0_ROCMFP4_COHERENT.gguf |

| Size | 4.2987 GiB (4,615,656,288 bytes) |

| BPW | 4.676 |

| ftype | Q4_0_ROCMFP4_COHERENT (102) |

| Source | BF16 GGUF (14.72 GiB) — lossless source, not a requantization |

| sha256 | fd9af463569509aca718f6ccacab388c74d8abe108511d5b6e9c00e3243b42b4 |

---

⛔ REQUIRES A PATCHED llama.cpp — STOCK WILL NOT LOAD THIS

Two independent reasons, both unavoidable:

1. bailingmoe3 is not in upstream llama.cpp. Support is still open in

PR #26608 (unmerged at time of writing).

1. Ling-3.0-tiny needs the Q-LoRA attention path. Its config sets q_lora_rank: 256

(q_a_proj → q_a_layernorm → q_b_proj). Several existing bailingmoe3 implementations were

written against Ling-3.0-flash, which has q_lora_rank: null and therefore no query

compression. On such a build, every GGUF of tiny fails — including the BF16 and Q4_K_M

ones — typically at missing tensor 'blk.0.ssm_f.weight', before the Q-LoRA gap is even

reached.

You need a build with both bailingmoe3 and its Q-LoRA path. The reference

implementation is the branch behind PR #26608

(aetherbird/llama.cpp, branch bailingmoe3-support). The ROCmFP4 quant types additionally

require a fork that implements them; upstream llama.cpp does not have Q4_0_ROCMFP4_*.

If your build loads Ling-3.0-flash but not tiny, you are missing the Q-LoRA path specifically.

⚠️ strings is not a capability check

We tested a second gfx1151 machine whose libllama.so contained **bailingmoe3 (60 matches),

ssm_f_a, and attn_q_a** — it looked fully capable. It still failed with the exact same

missing tensor 'blk.0.ssm_f.weight'.

Those symbols live in the tensor-name table. The fallback logic that maps ssm_f →

ssm_f_a, and the Q-LoRA branch itself, are separate code. **Grepping the binary tells you

nothing — attempt the load.**

---

All quant variants

Three builds of this model, all measured in one session on one box with one binary

(Ryzen AI MAX+ 395, gfx1151, ROCm 7.2.4, ROCmFPX-2809dc5) — so these rows are directly

comparable. Median of 3, warm-up discarded, otherwise-idle box.

| variant | ftype | size | bpw | decode (median) | range | repo |

|---|---|---|---|---|---|---|

| 4-bit COHERENT | 102 | 4.30 GiB | 4.67 | 104.04 | 104.00 – 104.24 | Ling-3.0-tiny-ROCmFP4-GGUF |

| 8-bit AGENT | 115 | 7.72 GiB | 8.40 | 88.82 | 88.80 – 88.83 | Ling-3.0-tiny-ROCmFPX-Q8_0-AGENT-GGUF |

| 8-bit plain | 111 | 7.62 GiB | 8.28 | 89.51 | 89.48 – 89.51 | Ling-3.0-tiny-ROCmFPX-Q8_0-GGUF |

⚠️ The 4-bit build is faster (104.04 vs ~89 tok/s) and 44% smaller. These 8-bit builds exist for accuracy headroom, not speed — pick them only if you need the extra precision.

What AGENT actually changes: it keeps far more tensors at true Q8_0 instead of the

packed 8-bit type — measured in these files, 135 tensors vs 2 tensors. On models with an

MTP draft head that raises draft acceptance and wins ~6%; these two models have no MTP head,

and here the two 8-bit builds are within noise of each other.

Measured results

Verified on two independent gfx1151 machines, using the model's official sampling

(temperature 0.6, top_p 0.95, top_k 20).

| | machine A | machine B |

|---|---|---|

| SoC | Ryzen AI MAX+ 395 (gfx1151) | Ryzen AI MAX+ 395 (gfx1151) |

| memory | 128 GB unified | 125 GB unified |

| ROCm | 7.2.4 | 7.13.0 |

| flags | -ngl 99 -c 4096 -fa on | -ngl 999 -c 32768 -fa on -fit off --no-mmap |

| loads | ✅ arch=bailingmoe3, 526 tensors | ✅ |

| 17 × 23 | ✅ 391 | ✅ 391 |

| capital of Japan | ✅ Tokyo | ✅ Tokyo |

| days in 2024 | ✅ 366 | ✅ 366 |

| reasoning separation | ✅ clean, in reasoning_content | ✅ |

| decode speed | 97.64 tok/s | 101.08 tok/s |

⭐ The build is portable across ROCm minor versions

The binaries were compiled against ROCm 7.2.4 and run unmodified on a ROCm 7.13.0 host —

all 9 Q4_0_ROCMFP4_* quant types still enumerated, model loads, 101 tok/s. Both hosts are

gfx1151. Useful if you have one build box and several Strix Halo machines: you can copy

llama-server + lib.so rather than rebuilding per host (set LD_LIBRARY_PATH to the

directory you copied them into).

Serving configuration that works

Long-running deployment on machine B (systemd, always-hot):

-dev ROCm0 -ngl 999 -fa on -c 32768 -fit off -np 1 --no-mmap --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20

plus LimitMEMLOCK=infinity, HSA_OVERRIDE_GFX_VERSION=11.5.1,

GGML_HIP_ENABLE_UNIFIED_MEMORY=1.

Verified on the running process, not just at launch: memlock unlimited (the 8 MB default

will hobble the model), n_ctx = 32768 actually granted — --fit is on by default and can

silently shrink context or push tensors to CPU, so -fit off and then confirm the number — and

zero "tensor override to CPU" lines in the log.

Per-tensor protection (audited in the finished file)

| tensor class | type |

|---|---|

| output.weight (LM head) | Q6_K |

| token_embd.weight | Q6_K |

| *_shexp shared experts (69) | Q8_0 |

| ffn_gate_inp router (23) | F32 |

| ssm_a, ssm_dt.bias (36) | F32 |

| ssm_conv1d_{q,k,v} (54) | F32 |

| norms (79) | F32 |

| routed experts, attention projections | 4-bit |

Why this matters. Tier _STRIX (105) protects attention K/V but not the LM head — on this

model's 157,184-token vocabulary that leaves every logit passing through a 4-bit tensor.

Tier 102 COHERENT carries Q6_K token embeddings, and the head/shared-expert protections above

were applied explicitly. Shared experts matter because they are dense — they process every

token, so their error is systematic rather than averaged across the 128 routed experts.

The recurrent/linear-attention state (ssm_a, ssm_dt, conv1d) is kept at F32: these are

float32 in the source model, and quantizing hybrid state is a known way to produce a model that

loads, runs, and emits fluent nonsense.

Size comparison (same source, same machine)

| build | size |

|---|---|

| BF16 | 14.72 GiB |

| Q4_K_M | 4.4926 GiB |

| this build | 4.2987 GiB |

---

What was NOT measured

Stated plainly so you can judge fitness for your use case:

• No perplexity run, and no quality A/B against Q4_K_M or BF16. The correctness checks

above are memorized-fact prompts — they are necessary but not sufficient, and a damaged model

can pass them.

• No long-context testing. All generations were short. The 32,768-token context was granted

and confirmed at load on machine B, but nothing exercised rope/KV behaviour at depth, and

nothing was run near the model's 131,072 ceiling.

• No tool-calling evaluation.
• MTP / speculative decoding untested — Ling-3.0-tiny has num_nextn_predict_layers: 0,

so it has no MTP layer to exercise.

---

Model

BailingMoeV3ForCausalLM / bailing_hybrid, GGUF arch bailingmoe3.

24 layers in a 3:1 stack of KDA (Kimi Delta Attention, 18 layers) and MLA

(Multi-head Latent Attention, 6 layers) · hidden 1536 · 128 routed experts, 8 active ·

shared experts · vocab 157,184 · q_lora_rank 256 · kv_lora_rank 512 · context 131,072.

Base model licence: MIT (inherited). All credit for the model itself goes to

inclusionAI.