---

license: mit

base_model:

• inclusionAI/Ling-3.1-flash

tags:

• gguf
• custom-quantization
• mixture-of-experts
• ling

language:

• en
• zh
• ko

---

Ling-3.1-flash VQ GGUF

Experimental VQ8 conversion of the official text-only

Ling-3.1-flash checkpoint,

pinned to a4ccd6b45c002ddb236f9cb77d52cc427482bf19.

This release uses a custom ling31_vq GGUF family. It requires a runtime that

implements FORMAT.md; ordinary llama.cpp, Ollama and Transformers

loaders do not implement this format.

ds4-dfm-rs runs it on one DGX Spark

(GB10): recipe v1 from 81eade8a (#92),

recipe v2 with #95 (the

long-context round), whose 709cf8df the measurements below use.

Selected recipe: ling31-vq8-mix8r13-60r12-mtp13-q4h-q8e-q4fg-v2. v2

replaces only body-nonexpert.gguf: the 126 KDA decay/gate projections are

Q4_K instead of block-scaled FP8 and the embeddings Q8_0 instead of BF16

(trunk 116.4 → 114.0 GB). It was derived from v1's part with the converter's

own quantizer; RECIPE.md and the

derivation receipt record how.

v1 (ling31-vq8-mix8r13-60r12-mtp13-q4h-v1) remains at revision e29aa7d8.

Conversion and file-integrity checks are complete, and all 71 GGUF parts are

published.

The original-model baseline completed 13 held-out cases; the final A/B quality

comparison and final selected MTP comparison were not completed. See

QUALITY.md for the recorded scope and limitations. The

assembly receipt records verified file sizes, hashes

and complete source-tensor accounting. This publication does not claim a final

model-quality pass.

Measured on DGX Spark (GB10)

!Ling-3.1-flash VQ 2K–64K prefill and decode on one DGX Spark

*Measured on this artifact (v2) with ds4-dfm-rs 709cf8df. One warm session

per process, 2,048-token incremental prefill and 128 forced greedy tokens per

frontier. Curves are per-frontier medians; bands are the observed min–max of

three fresh runs per variant.*

| Workload | Prefill tok/s | Decode tok/s |

|---|---:|---:|

| 2K–64K sweep, MTP off (median of three run means) | 774.1 | 17.70 |

| 2K–32K sweep, MTP off | 861.7 | 17.83 |

| 2K–32K sweep, two MTP drafts | 861.0 | 23.58 |

| 32K-token steps, frontiers 32K → 192K (single runs) | 862 → 376 | 17.56 → 16.22 |

| 123,809-token prompt, default 262K server | 616 | — |

v1 measured 778.8 / 14.86 (MTP off) and 859.8 / 21.15 (two drafts) at

a64e323b (#93).

• Decode gains 16% at 2K and 20% at 64K over v1: Q4_K decay/gate remove

1.85 GB of each token's 12.6 GB of weight reads, and ds4-dfm-rs now keeps

MLA latent rows as FP8 (641 instead of 1,152 bytes per token and layer).

Incremental prefill falls 35% from 4K (964.9) to 64K (625.8): the nine MLA

layers expand each 2,048-key segment of the latent cache in every step.

• MTP drafts greedy requests only; thinking requests, which sample at

temperature 1.0, decode without drafts. Two drafts measured best.

• ds4-server defaults to the native 262,144-token context: two persistent

banks at 2,048-row prefill chunks with v2 (one with v1), or one bank with

two MTP drafts (--mtp-mode on --mtp-draft 2). After a restart, the next

turn of a 124K-token conversation restored 123,931 of 123,957 tokens from

disk KV in 3.7 s.

• CUDA 13.4, sm_121a, 300–2,200 MHz clock policy; idle 299–305 MHz before

each run and 2,184 MHz in one sample right after a run (not sampled during

the sweep).

Model and precision

The original model has 72 trunk layers: 63 KDA and nine gated MLA blocks. Layers

0–3 use dense FFNs; layers 4–71 have 512 routed experts and select eight per

token. Hidden width is 4096, expert intermediate width is 1280, and the

vocabulary contains 157184 tokens. Layer 72 is a separate MLA/MoE MTP block.

There is no vision encoder, Engram, PLE, mHC or sparse attention indexer in this

checkpoint.

Routed expert weights use vectors of eight consecutive input-axis weights,

one independently trained E4M3 codebook shared by all expert projections in

each layer, and FP16 row RMS gains. The selected mixed allocation and its

screening evidence are recorded in RECIPE.md and recipe.json.

Trunk layers 4, 5, 6, 9, 12, 43, 57 and 58 use K8192; the other 60 routed

trunk layers use K4096. MTP layer 72 uses a separate K8192 codebook.

K4096 uses 12-bit indices (1.5 index bits/weight); K8192 uses 13-bit indices

(1.625 index bits/weight). Gains, codebooks and container overhead are additional.

The 13th bit is a separate bit plane. Each K4096 codebook was trained separately;

it is not a truncated K8192 codebook.

The conservative nonexpert policy quantizes 504 large projections to Q4_K and

uses a separately evaluated Q4_K output head. Recipe v2 also stores the 126

KDA decay/gate (f/g) projections as Q4_K, encoded from the FP8 values times

their block scales rounded to BF16, and the embeddings as Q8_0. Router, bias,

norms, b-projection, convolution, recurrence controls, MLA kv_b and head gates

retain their original mixed precision. MTP nonexpert weights

retain their original precision. No zchain, GR/RB correction, LoRA, QAT,

pruning or distillation is applied. All quantized weights originate directly

from the pinned original checkpoint.

Files and optional MTP

release.json is the loading entrypoint. These are independent GGUF parts,

not standard numbered GGUF splits; do not concatenate them.

| Feature | Required parts |

| --- | --- |

| Trunk, default | body-nonexpert.gguf and expert parts for layers 04–71 |

| Trunk with MTP | All trunk parts, mtp-native.gguf, and layer-72-experts.gguf |

body-nonexpert.gguf contains the attention and dense FFN weights for layers

0–3, plus the other trunk layers' nonexpert weights, output head and embeddings.

The optional MTP experts use their own K8192 codebook and FP16 row gains.

MTP is an approximately 8.17B-parameter block, not merely an extra output head.

With MTP disabled, select the trunk feature before mapping or uploading parts;

there must be no MTP GPU weights, repacking, cache or workspace allocation.

The selected MTP parts contain no duplicate original expert weights.

tensor-manifest.json maps each of the 214025 original tensors exactly once

to its preserved or quantized output and lists the generated codebooks.

files.json and SHA256SUMS identify the released file contents. Configuration,

tokenizer, chat template and special-token files are unchanged original assets.

See SOURCE.md for provenance and licensing.

The unchanged config.json describes the original FP8 checkpoint; the release

recipe and tensor manifest define the stored GGUF precisions.

Numerical and quality scope

The numerical reference separates original-checkpoint inference (A) from

inference using the exact stored quantized bytes (B). Original FP8 values use

their multiplicative F32 block scales, then the declared BF16 weight boundary.

VQ and Q4_K stored values are independently reconstructed before the same

boundary. Router and recurrence controls follow the documented F32 contract;

SwiGLU clamps follow the pinned serving implementations, including the distinct

shared/routed layer lists and unclamped MTP layer.

Original-model baseline A completed all 13 held-out cases: 12 generations ended

with normal EOS, and the Korean story reached its 768-token cap without a

completed ending. The code answer passed six functional cases but included

Markdown fences despite a code-only instruction. These are baseline findings;

they do not measure the released VQ model's quality. Final matched A/B

NLL/PPL, KL, top-1 agreement, generation-quality results and the final selected

MTP comparison were not completed. Screening measurements are not a final

model-quality certificate. QUALITY.md summarizes the evidence.

A runtime screen with the ds4-dfm-rs CUDA backend teacher-forced the same 13

held-out cases on reference A's token ids: NLL 1.043 (v1 1.046) against A's

1.117 over the same 642 targets, higher than A in 7 of 13 cases, with 86.8%

top-1 agreement. The targets are fixture text, so neither model is favoured;

13 cases do not measure a quality gain, and this screen is not the final A/B.

Against v1, v2 moved the 32K–192K frontier logits by KL 0.00003–0.0322 with

the same argmax, the same order as switching between 1,024- and 2,048-row

prefill chunks (0.0002–0.0815 at 32K–128K); teacher-forced NLL after 32K, 64K

and 128K prompts was 2.071 against v1's 2.080.

FP8 activation execution, gain-after-dot kernels and direct Q4_K serving

arithmetic are outside this reference comparison. BF16 reconstruction in the

quality reference does not establish serving speed or dequantization cost.

The original EOS is 156895 (<|role_end|>); padding is 156892. The original

thinking and tool-call tokens and chat template are preserved. Greedy

diagnostics are distinct from official sampling with temperature 1.0,

top-p 0.95 and top-k 20.

Memory and context

The assembled 71 GGUF parts occupy 115760613632 bytes (107.810473 GiB),

including MTP. The trunk accounts for 113960788096 bytes and optional MTP adds

1799825536 bytes (v1: 118214943744 bytes in all). These are verified written sizes, including container

headers, scales, codebooks, gains and padding. Tokenizer, documentation and

audit manifests are additional files.

memory-budget.json models v1's part sizes,

batch-one KDA states and BF16 latent MLA caches. It compares conditional

residency scenarios against a historical DGX Spark MemTotal of 130596163584

bytes (121.627155304 GiB), which has not been rechecked on the target machine.

The budget reserves 4 GiB for OS/driver, 3 GiB for shared workspace and 1 GiB

for loading.

| Context tokens, batch one | Modeled peak with MTP | Remaining historical MemTotal |

| --- | --- | --- |

| 32768 | 118.705438 GiB | 2.921718 GiB |

| 65536 | 119.057000 GiB | 2.570155 GiB |

| 262144 | 121.166375 GiB | 0.460780 GiB |

These are budget calculations, not measured GB10 peaks or supported-context

claims. Measured with ds4-dfm-rs, a 262,144-token session at 2,048-row

chunks takes 3.12 GiB (1.41 GiB FP8 latent KV, 276 MiB KDA state, 186 MiB

controls, 1.26 GiB workspace), each further bank 1.68 GiB, and the first

requests add 0.41–0.48 GiB as CUDA loads kernel images and workspaces. Resident repacking, duplicate CPU

weights and persistent BF16 expansion require additional memory. MTP

draft/verify workspace must fit the shared allowance. All active compressed weights are counted as resident.

Ordinary layer or expert offloading would need separately measured transfer

and compute costs; this budget includes no offload savings.

The unchanged configuration specifies max_position_embeddings=262144 and

rope_scaling=null. The upstream model card's 1M-context statement does not

establish a supported context for this conversion. Neither storage-budget

scenarios nor short numerical evaluations qualify long-context serving.

ds4-dfm-rs implements the GB10 kernels, banks, disk KV and MTP drafting

measured above and screened logits and NLL to 192K; long-running serving and

long-context quality are not qualified.