---

license: mit

library_name: gguf

pipeline_tag: text-generation

inference: false

base_model: inclusionAI/Ling-3.0-tiny

base_model_relation: quantized

model_name: Ling-3.0-tiny GGUF

quantized_by: Mike0021

tags:

- gguf

- llama.cpp

- bailingmoe3

- mixture-of-experts

- quantized

- reasoning

- conversational

---

Ling-3.0-tiny GGUF

Unofficial GGUF conversion and importance-matrix quantizations of

inclusionAI/Ling-3.0-tiny,

created from immutable source revision

a2ee06c0.

No fine-tuning, merging, or other parameter training was performed. The

original model documentation, intended use, benchmark claims, and limitations

remain authoritative.

> Experimental runtime requirement

>

> As of 2026-08-11, BailingMoE3 support remains unmerged in upstream

> llama.cpp. These files were converted and validated with

> PR #26608 at exact commit

> d8d8625.

> This includes the Q-LoRA path required by Ling-3.0-tiny

> (q_lora_rank=256) from

> 517b4675

> and the pinned multi-argument tool-parser fix

> 0266ebca.

> Stock or older llama.cpp binaries and other GGUF

> runtimes may reject this architecture or produce incorrect output until they

> incorporate equivalent support.

Preserved model facts

• BailingMoeV3 hybrid KDA/MLA sparse MoE, 526 GGUF tensors
• 7,893,392,800 parameters total; approximately 1.3B active per token
• 24 layers; 128 routed experts, 8 selected per token, plus 1 shared expert
• Q-LoRA rank 256 and KV-LoRA rank 512
• Native configured context: 131,072 tokens
• Embedded tokenizer and source chat template
• No NEXTN/MTP layers (num_nextn_predict_layers=0)

The source identifies itself as Transformers model_type=bailing_hybrid with

BailingMoeV3ForCausalLM; the pinned converter intentionally maps that model

to GGUF general.architecture=bailingmoe3. This is not a model-family

mismatch.

The original card's 256K command uses an external YaRN/runtime override. This

release preserves the checkpoint's native 131,072-token configuration and does

not claim validated 256K operation. Do not enable MTP speculative decoding for

this Tiny checkpoint.

Files and recommendations

| File | Quant | Size | Matrix | Suggested use |

|---|---:|---:|:---:|---|

| Ling-3.0-tiny-BF16.gguf | BF16 | 14.72 GiB | No | Exact GGUF reference/requantization source |

| Ling-3.0-tiny-Q8_0.gguf | Q8_0 | 7.83 GiB | No | Highest-fidelity quantized option |

| Ling-3.0-tiny-Q6_K.gguf | Q6_K | 6.05 GiB | Yes | Quality-first practical choice |

| Ling-3.0-tiny-Q5_K_M.gguf | Q5_K_M | 5.25 GiB | Yes | Recommended quality/size balance |

| Ling-3.0-tiny-Q4_K_M.gguf | Q4_K_M | 4.49 GiB | Yes | Recommended lower-memory default |

| Ling-3.0-tiny-Q4_K_S.gguf | Q4_K_S | 4.24 GiB | Yes | Smaller K-quant alternative |

| Ling-3.0-tiny-IQ4_XS.gguf | IQ4_XS | 3.99 GiB | Yes | Most compact 4-bit option |

| Ling-3.0-tiny-Q3_K_M.gguf | Q3_K_M | 3.58 GiB | Yes | Larger K-quant 3-bit tier |

| Ling-3.0-tiny-IQ3_M.gguf | IQ3_M | 3.31 GiB | Yes | Smaller 3-bit tier |

| Ling-3.0-tiny-IQ2_M.gguf | IQ2_M | 2.52 GiB | Yes | Extreme compression; substantial loss |

| Ling-3.0-tiny-imatrix.gguf | Auxiliary | 41.98 MiB | — | Reproducing importance-aware quants |

If memory permits, prefer Q6_K or Q8_0 for fidelity. Q5_K_M is the

quality-oriented general recommendation; Q4_K_M is the lower-memory default.

IQ3_M and IQ2_M are specialized memory-constrained choices; the measured loss

at IQ2_M is large enough that it should not be a default. File size is not

total runtime memory: context length, state/KV caches, backend, and GPU offload

add overhead. IQ backend support varies, so use the pinned runtime until

equivalent BailingMoE3 support lands elsewhere.

Checksums are in SHA256SUMS.

Download and run

hf download Mike0021/Ling-3.0-tiny-GGUF \
  --include "Ling-3.0-tiny-Q5_K_M.gguf" \
  --local-dir ./models

Build the tested unmerged runtime (review the PR before running it):

git clone --filter=blob:none https://github.com/ggml-org/llama.cpp.git
git -C llama.cpp fetch origin refs/pull/26608/head:pr-26608
git -C llama.cpp checkout d8d862521e9ad842f2b47f3b392b039317782aa0
cmake -S llama.cpp -B llama.cpp/build -DGGML_CUDA=ON -DGGML_NATIVE=OFF
cmake --build llama.cpp/build --config Release --parallel

For a CPU-only build, omit -DGGML_CUDA=ON. This server example deliberately

starts at 8K context to keep memory moderate:

./llama.cpp/build/bin/llama-server \
  -m ./models/Ling-3.0-tiny-Q5_K_M.gguf \
  --alias ling-3.0-tiny --host 127.0.0.1 --port 8080 \
  --jinja -c 8192 -ngl 999

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ling-3.0-tiny",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "stream": false
  }'

These sampling settings follow the original model's recommendations. Thinking

is enabled by the embedded source chat template by default.

To disable thinking in the pinned server, pass

"chat_template_kwargs":{"enable_thinking":false} in the request. Keep

--jinja enabled so the embedded template is applied.

The pinned runtime logs special_eos_id is not in special_eog_ids while

loading this tokenizer. The raw arithmetic reference stopped on token 156895

in Transformers, and Q4_K_M server stop behavior was tested as described

below, but the warning is preserved here because it has not yet been resolved

upstream.

Conversion provenance

| Item | Value |

|---|---|

| Source | inclusionAI/Ling-3.0-tiny@a2ee06c0f2de5b171701aee7f73f70a1da75483b |

| Source weights | 32 safetensors shards, 15,787,992,416 bytes |

| Converter/runtime | aetherbird/llama.cpp@d8d862521e9ad842f2b47f3b392b039317782aa0 (upstream PR #26608) |

| Conversion | BF16 GGUF, then every quant directly from BF16 |

| Detailed provenance | conversion_manifest.json |

| Source shard hashes | source-safetensors.sha256 |

| Core reproduction commands | REPRODUCE.md |

Importance-matrix calibration

Importance-aware files used two complementary, pinned calibration sources.

The primary corpus was

lemon07r/bartowski-imatrix-v5-semantic

at revision a306f203ee4323e0afe846ae02c2daafe17384d9. Its 2,075 semantic

samples span 13 languages and include code, math, science, dialogue, and Q&A,

which is substantially broader than English-only WikiText calibration.

An additive second pass used combined_all_micro.parquet from

eaddario/imatrix-calibration

at revision e87ed55dcba9d9c3a3e41539f3e728e981b1daa4. This MIT-licensed

mixture adds multilingual text plus tool-use, math, and code prompts. It was

added because the first pass left one routed expert unobserved in one layer;

the release gate requires every routed-expert slot to have a nonzero count.

• Input: bartowski-imatrix-v5-semantic.txt
• SHA-256: ff879b5a748f822ef539e43c596a3f44ab922f0295ee209d4220d9f86e86a063
• 1,496,006 bytes; 6,318 serialized lines
• Supplement parquet SHA-256:

94389921e1f67b180a99de28c3090b41ce6f1960eb13abad21b7eba7cbe11b26

• Extracted supplement SHA-256:

fdb2d41abf04a2fb207502741a561a5a9ab385eb0c44a450eae676c410955946

(1,008,653 bytes; 3,130 serialized lines)

• Context / batch / ubatch: 4096 / 4096 / 512
• Complete 4,096-token chunks processed: 162

(663,552 tokens); 5,338 trailing tokens excluded

• Matrix entries: 332
• Per-expert count values: 8,832
• Routed-expert slots with zero observations: 0

The matrix is the modern GGUF imatrix format. It contains 69 expert-count

vectors of length 128 (8,832 layer/tensor expert slots); “zero” is measured

over those slots, not over 128 globally unique expert IDs. Output-tensor

statistics were intentionally not collected: the pinned llama.cpp imatrix

documentation says it is typically better not to use importance statistics

when quantizing output.weight, and therefore defaults --process-output to

false.

Observed per-slot counts ranged from 16 to 326,023 (median 33,514); a

distribution summary and the lowest-count slots are recorded in

validation/imatrix.json.

The final matrix SHA-256 is

e8b15d131f9ce294f922c5c387f7a69829c12100d6a35bb1635a2b859083c3f0.

llama-quantize embeds only one quantize.imatrix.dataset scalar, so the

importance-aware model files name the primary corpus even though the final

matrix contains both ordered passes. The manifest is the authoritative record

of the two-source lineage. It also records the absolute paths embedded by the

quantizer; changing those paths can preserve tensor values while changing the

GGUF file hash.

The corpus was used only to collect activation statistics. It was not used to

train or fine-tune the model and is not an evaluation set.

Held-out validation

Validation used the separate WikiText-2 test file from

ggml-org/ci@927b3642933080f1b0e811e2f916e14c292992f9; this file was not

used for imatrix collection. Content-level uniqueness from all calibration

material or from the model's original pretraining data is not asserted. The

extracted wiki.test.raw SHA-256 is

173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08.

PPL and BF16-relative KLD used 32 fixed sequential chunks at

context/batch/ubatch 512, scoring

8,160 held-out tokens. Exact commands are in

REPRODUCE.md, and machine-readable results are under

validation/.

| Artifact | Loads | Greedy raw vs HF BF16 | PPL ± SE | ΔPPL | Mean KLD ± SE (nats) |

|---|:---:|:---:|---:|---:|---:|

| BF16 self | Pass | Exact | 11.901303 ± 0.415179 | +0.033176 | 0.000000 ± 0.000000 |

| Q8_0 | Pass | Exact | 11.812842 ± 0.410345 | −0.055285 | 0.011688 ± 0.000329 |

| Q6_K | Pass | Exact | 11.873857 ± 0.413599 | +0.005730 | 0.023357 ± 0.000625 |

| Q5_K_M | Pass | Exact | 12.087854 ± 0.422594 | +0.219727 | 0.053244 ± 0.001318 |

| Q4_K_M | Pass | Exact | 12.651529 ± 0.447483 | +0.783402 | 0.130069 ± 0.003051 |

| Q4_K_S | Pass | Exact | 12.608386 ± 0.443531 | +0.740259 | 0.138631 ± 0.003234 |

| IQ4_XS | Pass | Exact | 12.640906 ± 0.445231 | +0.772779 | 0.155524 ± 0.003489 |

| Q3_K_M | Pass | Exact | 13.649613 ± 0.484819 | +1.781486 | 0.301154 ± 0.006362 |

| IQ3_M | Pass | Exact | 12.967071 ± 0.446764 | +1.098944 | 0.312063 ± 0.006496 |

| IQ2_M | Pass | Exact | 16.362374 ± 0.564546 | +4.494247 | 0.696147 ± 0.011718 |

These tests measure conversion and quantization behavior, not general model

capability or safety. Results are comparable only under the documented

tokenizer, context, chunk, and pinned-runtime settings. The stored BF16

reference has PPL 11.868127 ± 0.412222. BF16 self-comparison establishes the

uint16 stored-log-probability/backend resolution; mean KLD rounded to 0.000000

nats in this run. Small negative ΔPPL values, such as Q8_0, are within sampling

uncertainty and do not mean the quant is better than BF16.

“Loads” means the pinned runtime completed its tensor integrity/load check and

a graph evaluation. “Greedy raw vs HF BF16” compares a deterministic 12-token

continuation against a separately generated Transformers BF16 reference. The

validator binds both runtimes to the exact same full prompt; all ten artifacts

matched this one shallow case exactly. This is a conversion smoke test, not a

claim that quantized logits or arbitrary generations equal BF16. All six

tokenizer test cases, including Chinese, code, whitespace, multilingual text,

and special tokens, matched Transformers token IDs exactly.

Q6_K contains six Q8_0 fallbacks because those narrow MLA tensors cannot use

the requested block width. The 3-bit and 2-bit files likewise contain exactly

six documented MLA fallbacks. Their complete tensor-type inventories are in

the structure reports and manifest.

Matrix ablation

A direct Q4_K_M A/B against a temporary no-matrix quant gave mixed evidence.

The matrix lowered the mean KLD point estimate from 0.131547 to 0.130069 nats

and raised the same-top-token point estimate from 84.596% to 85.221%, while

PPL moved from 12.357816 to 12.651529. This is not presented as a universal

quality gain; the broader calibration coverage and those KLD/same-top point

estimate shifts motivated retaining the matrix build. See

kld-Q4_K_M-ab.json.

Fixed multiple-choice collapse screen

The pinned mmlu-validation.bin contains 1,548 four-choice tasks. A fixed

seed-1 subset of 500 was used as a regression/collapse check, not as a model

capability benchmark. The tool's log says “TruthfulQA,” but the supplied input

is the pinned MMLU validation binary (SHA-256

470af3a74eccacfaf6f43b08aabf510f61e6c92fe20d17241ded934151e225fa).

| Artifact | Accuracy ± SE |

|---|---:|

| BF16 | 38.2% ± 2.1751% |

| Q5_K_M | 39.0% ± 2.1835% |

| Q4_K_M | 38.8% ± 2.1814% |

| Q4_K_S | 39.0% ± 2.1835% |

| IQ4_XS | 37.2% ± 2.1637% |

| Q3_K_M | 37.8% ± 2.1707% |

| IQ3_M | 37.8% ± 2.1707% |

| IQ2_M | 34.8% ± 2.1324% |

Random chance was 25.0% ± 1.9384%. Q8_0 and Q6_K were not run through this

auxiliary screen; their held-out KLD results are the stronger fidelity evidence.

Long-context and server checks

BF16, Q4_K_M, and the most aggressive IQ2_M completed a one-chunk 32,768-token

perplexity/prefill evaluation at batch 4,096: respectively 23.3709, 25.7803,

and 34.6812 PPL. Other artifacts were validated at context 512. The checkpoint's

native 131,072-token limit and the external 256K YaRN configuration were not

exercised.

Q4_K_M was also tested through llama-server --jinja. Thinking-disabled and

thinking-enabled requests both stopped normally, the latter exposed separate

reasoning content, a Chinese prompt returned 巴黎, and a required

tool request produced get_weather with both location=Paris and

unit=celsius arguments and finish_reason=tool_calls. These server results

apply to Q4_K_M; they are not generalized to every quant.

Rejected candidates

Two generated candidates were deliberately not published. IQ4_NL was only

28,606,464 bytes smaller than Q4_K_S while its KLD rose from 0.138631 to

0.149734. MXFP4_MOE passed an exact 69-tensor routed-expert whitelist, but at

4,718,248,800 bytes and 0.267021 KLD it was larger and much less faithful than

Q4_K_S. On the tested RTX PRO 4500 Blackwell it improved 512-token prompt

throughput by 17.9% but reduced 128-token generation throughput by 8.2%.

Full measurements are in

rejected-candidates.json.

As a post-hoc independent cross-check, the canonical BF16 and Q8_0 SHA-256

values exactly match

bloomer010/Ling-3.0-tiny-GGUF@598201.

That repository was not used as a weight source.

Limitations and attribution

• Runtime support is experimental and tied to an unmerged llama.cpp revision.
• Quantization can change factuality, reasoning, tool-call formatting, and

multilingual behavior; validate the chosen file on your workload.

• Long contexts add substantial memory and were not exhaustively exercised for

every artifact.

• No new safety evaluation was performed. The source model's limitations and

acceptable-use considerations still apply.

• This is an unofficial conversion, not endorsed by InclusionAI, Hugging Face,

or llama.cpp maintainers.

The source card declares the MIT license. Original authorship belongs to

InclusionAI; this repository provides an unofficial format conversion by

Mike0021.