GraySoft
Download guIDE →
Projects Models Compare Cloud benchmarks FAQ Download guIDE →
Model Intelligence Sheet

wepiqx/Ling-3.0-tiny-ASHQ1-GGUF overview

Ling 3.0 tiny — ASHQ1 Quantization Top Down ASHQ1 quantization of Ling 3.0 tiny https://huggingface.co/inclusionAI/Ling 3.0 tiny 7.9B total / 1.3B active, 128 …

ggufmoehybrid-linear-attentionkdamlalingreasoningquantizedquantizationllama-cppimatrixhybrid-quantizationASHQ1priority-queuetop-downtext-generationenzhbase_model:inclusionAI/Ling-3.0-tinybase_model:quantized:inclusionAI/Ling-3.0-tinylicense:apache-2.0endpoints_compatibleregion:usconversational

Runs locally from ~4.75 GB disk (8 GB VRAM class GPUs with llama.cpp / guIDE).

Downloads
0
Likes
1
Pipeline
text-generation
Author
wepiqx

Repository Files & Downloads

2 GGUF files detected
Direct downloads for local inference
FileTypeQuantizationSizeLink
Ling-3.0-tiny-BF16-ASHQ1-5000.ggufGGUFBF164.75 GBDownload
Ling-3.0-tiny-BF16-TOPDOWN-6000.ggufGGUFBF165.87 GBDownload

Model Details

Model IDwepiqx/Ling-3.0-tiny-ASHQ1-GGUF
Authorwepiqx
Pipelinetext-generation
Licenseapache-2.0
Base modelinclusionAI/Ling-3.0-tiny
Last modified2026-09-06T10:34:28.000Z

Model README

---

license: apache-2.0

language:

  • en
  • zh

base_model: inclusionAI/Ling-3.0-tiny

pipeline_tag: text-generation

library_name: gguf

tags:

  • moe
  • hybrid-linear-attention
  • kda
  • mla
  • ling
  • reasoning
  • quantized
  • quantization
  • gguf
  • llama-cpp
  • imatrix
  • hybrid-quantization
  • ASHQ1
  • priority-queue
  • top-down

---

Ling-3.0-tiny — ASHQ1 Quantization (Top-Down)

ASHQ1 quantization of Ling-3.0-tiny (7.9B total / 1.3B active, 128 routed + 1 shared experts, KDA+MLA hybrid attention) with the top-down allocator: every tensor starts at F16 and is greedily downgraded cheapest-loss-first until the budget fits.

MoE-specific handling: routers (ffn_gate_inp, exp_probs_b) pinned at F16 outside the budget, shared experts mapped to high tiers (always active), routed experts graded by depth (early layers Q4_K → late layers Q6_K/Q8_0).

> Note: File names contain "BF16" for HuggingFace parser compatibility — these are ASHQ1 quants, not BF16.

Quants

| File | Size | PPL (wiki.test.raw, ctx 1024) | Description |

|:-----|:----:|:-----------------------------:|:------------|

| Ling-3.0-tiny-BF16-ASHQ1-5000.gguf | 4861 MiB | 13.0681 ± 0.1051 | Bottom-up, best of the duel — smaller and better |

| Ling-3.0-tiny-BF16-TOPDOWN-5000.gguf | 4999 MiB | 13.1094 ± 0.1054 | Top-down challenger (local only) |

| Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf | 6013 MiB | 13.0001 ± 0.1047 | Top-down, first MoE quant |

| Ling-3.0-tiny-BF16-TOPDOWN-6000-v7.1.gguf | 6016 MiB | 12.9802 ± 0.1045 | Top-down + phase-2 slack filling (local only) |

| Ling-3.0-tiny-i1-Q6_K.gguf | 6198 MiB | 13.0123 ± 0.1049 | Uniform Q6_K with imatrix (reference, local only) |

Bottom-up vs top-down (@5000, same budget)

Two opposite allocations, same PPL within noise (Δ=0.04, σ=0.10):

| Tier | Bottom-up (4861 MiB) | Top-down (4999 MiB) |

|:----:|:--------------------:|:-------------------:|

| F16 | 215 / 9.7 MiB | 126 / 222.8 MiB |

| Q4_K | 43 / 2271 MiB | 210 / 2059 MiB |

| Q5_K | 30 / 2036 MiB | 36 / 2304 MiB |

| Q6_K | 8 / 94 MiB | 14 / 109 MiB |

| Q8_0 | 230 / 432 MiB | 140 / 285 MiB |

Bottom-up spreads upgrades wide (241 small tensors at Q8_0); top-down keeps the most precious tensors at F16 (incl. top-importance shared experts) and crushes the rest to the floor. Lesson: PPL can't separate the two paths — the bit distribution differs radically, the quality doesn't. Finer-grained tasks (HumanEval, MMLU) may.

Speed

MoE sparsity pays off: only 1.3B of 7.9B params active per token. On a GTX 1070 (8 GB) the quant runs at ~76 t/s mixed load and fits the full 131072-token context without problems.

Distribution (@6000, top-down)

| Tier | Tensors | Notes |

|:----:|:-------:|:------|

| F16 | 248 | pinned norms/routers + untouched high-importance |

| Q4_K | 183 | early routed experts, floors |

| Q5_K | 23 | incl. output/token_embd pins |

| Q6_K | 41 | late routed experts |

| Q8_0 | 31 | shared experts, MLA factors |

Usage

llama.cpp

llama-cli \
  -m Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf \
  --jinja \
  -ngl 99 \
  -c 8192

Recommended sampling: temperature 1.0, top_p 0.95, top_k 20.

Ollama

FROM ./Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf

PARAMETER num_ctx 8192
PARAMETER temperature 1.0
PARAMETER top_k 20
PARAMETER top_p 0.95
ollama create ling-tiny-topdown-6000 -f Modelfile
ollama run ling-tiny-topdown-6000

Quantization Config (@6000, top-down)

Generated by ASHQ1 top-down. Reproduce with:

python main.py --model Ling-3.0-tiny-bf16.gguf --imatrix Ling-3.0-tiny-imatrix.gguf --size 6000 --top-down --run

<details>

<summary>llama-quantize arguments (click to expand)</summary>

--output-tensor-type Q5_K
--token-embedding-type Q5_K
--tensor-type "(blk|BLK)\.(23)\.ffn_down_exps=Q8_0"
--tensor-type "(blk|BLK)\.(0)\.ffn_gate=Q8_0"
--tensor-type "(blk|BLK)\.(0)\.ffn_up=Q8_0"
--tensor-type "(blk|BLK)\.((?:2|3|4|5|6|7|8|9|10|11|12|13))\.ffn_down_shexp=Q8_0"
--tensor-type "(blk|BLK)\.(1)\.ffn_down_shexp=Q6_K"
--tensor-type "(blk|BLK)\.((?:6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21))\.attn_output=Q8_0"
--tensor-type "(blk|BLK)\.((?:17|18|19|20|21|22))\.ffn_down_exps=Q6_K"
--tensor-type "(blk|BLK)\.(0)\.ffn_down=Q4_K"
--tensor-type "(blk|BLK)\.((?:9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_exps=Q6_K"
--tensor-type "(blk|BLK)\.([2-5])\.attn_output=Q6_K"
--tensor-type "(blk|BLK)\.((?:9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_up_exps=Q6_K"
--tensor-type "(blk|BLK)\.(1)\.ffn_up_exps=Q4_K"
--tensor-type "(blk|BLK)\.(1)\.ffn_gate_exps=Q4_K"
--tensor-type "(blk|BLK)\.((?:14|15|16|17|18|19|20|21|22|23))\.ffn_down_shexp=F16"
--tensor-type "(blk|BLK)\.((?:12|13|14|15|16))\.ffn_down_exps=Q5_K"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11))\.ffn_down_exps=Q4_K"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.exp_probs_b=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_inp=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_beta=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_k=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_kv_a_mqa=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_a=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_shexp=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_b=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_q=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_f_a=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_v=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_g_a=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_up_shexp=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_gate=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_k_b=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_v_b=F16"
--tensor-type "(blk|BLK)\.((?:22|23))\.attn_output=F16"
--tensor-type "(blk|BLK)\.([2-8])\.ffn_up_exps=Q5_K"
--tensor-type "(blk|BLK)\.([2-8])\.ffn_gate_exps=Q5_K"
--tensor-type "(blk|BLK)\.([0-1])\.attn_output=Q5_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_k=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_a=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_dt=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.attn_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_v=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_q=Q4_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_a_norm=Q4_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_kv_a_norm=Q4_K"
--tensor-type ".*output_norm.*=Q4_K"

</details>

Notes

  • Source BF16: inclusionAI/Ling-3.0-tiny (24 layers, bailingmoe3 arch, no MTP)
  • Imatrix: custom Ling-3.0-tiny imatrix
  • MoE: 128 routed experts (8 active) + 1 shared expert per token, expert FF 512 (no K-quant padding needed)
  • Quantization and tuning by wepiqx
  • Built with llama.cpp
  • License: Apache-2.0 (upstream)

Run wepiqx/Ling-3.0-tiny-ASHQ1-GGUF with guIDE

Download guIDE — the AI-native code editor with local LLM inference and 69 built-in tools.

Download guIDE → · Browse 524k+ models · Compare models

Source: Hugging Face · Compare models