Baekpica/Ling-3.0-flash-VL-Mixed-Quant-GGUF overview
Ling 3.0 flash VL Mixed Quant GGUF One accelerator resident variant, MQ Q5 KDA VIT BF16 https://huggingface.co/Baekpica/Ling 3.0 flash VL Mixed Quant GGUF/tree…
Runs locally from ~837.5 MB disk (4 GB VRAM class GPUs with llama.cpp / guIDE).
Repository Files & Downloads
| File | Type | Quantization | Size | Link |
|---|---|---|---|---|
| MQ-Q5-KDA-VIT-BF16/Ling-3.0-flash-VL-MQ-Q5-KDA-VIT-BF16-00001-of-00003.gguf | GGUF | Q5 | 27.71 GB | Download |
| MQ-Q5-KDA-VIT-BF16/Ling-3.0-flash-VL-MQ-Q5-KDA-VIT-BF16-00002-of-00003.gguf | GGUF | Q5 | 27.93 GB | Download |
| MQ-Q5-KDA-VIT-BF16/Ling-3.0-flash-VL-MQ-Q5-KDA-VIT-BF16-00003-of-00003.gguf | GGUF | Q5 | 21.88 GB | Download |
| MQ-Q5-KDA-VIT-BF16/mmproj-Ling-3.0-flash-VL-BF16.gguf | GGUF | Q5 | 837.5 MB | Download |
Model Details
| Model ID | Baekpica/Ling-3.0-flash-VL-Mixed-Quant-GGUF |
|---|---|
| Author | Baekpica |
| Pipeline | image-text-to-text |
| License | mit |
| Base model | inclusionAI/Ling-3.0-flash-VL |
| Last modified | 2026-09-17T01:00:48.000Z |
Model README
---
license: mit
base_model: inclusionAI/Ling-3.0-flash-VL
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: gguf
tags:
- gguf
- mixed-quant
- bailingmoe3
- ling-3.0-flash-vl
- kda
- linear-attention
- dgx-spark
- ds4
---
Ling-3.0-flash-VL Mixed-Quant GGUF
> One accelerator-resident variant, MQ-Q5-KDA-VIT-BF16, built directly from the pinned BF16 checkpoint. The 124.4B-parameter language tower costs 83,220,797,312 bytes / 77.5054 GiB at 5.3512 effective BPW, and the BF16 vision tower and projector add 878,174,368 bytes / 0.8179 GiB for a complete multimodal residency of 84,098,971,680 bytes / 78.3233 GiB.
>
> The recipe is asymmetric by tensor role, not by a uniform tier. Every tensor on the recurrent critical path — the KDA decay/gate projections, both attention families, the always-active shared expert, the leading dense FFN, and the entire vision tower — is carried at BF16, and the logit head is BF16 as well. The whole quantization budget is spent inside the 512-way routed experts, which are 97.1% of the parameters.
>
> No importance matrix is used. The assignment is deterministic and reproducible from the source checkpoint alone; there is no calibration corpus to trust and no imatrix binary to ship.
>
> DGX Spark runtime qualification has now run. ds4-dfm-rs serves this artifact as its eleventh model family: text and still-image chat, two concurrent persistent banks, exact and partial prefix reuse, and disk KV, at the native 131,072-token context. That is the serving surface of the Qwen3.8 Flash Next MQ-Q5 artifact this recipe was sized against, minus MTP: bailingmoe3 has no predictor block to run. See runtime support for what was measured and what was not.
>
> Update 2026-09-17 — CUDA prefill and decode optimization (PR #48–PR #50, main f35dbeb): On one DGX Spark the 8,192+64 ds4-bench went 1,142 → 1,889 tok/s prefill (+65%) and 19.54 → 24.65 tok/s decode (+26%) through tensor-core absorbed-MLA prefill attention, 4,096-row prefill chunks, one RMSNorm→BF16 convert per packed GEMM and a row-stable n=1 BF16 warp GEMV (PR #48). PR #49 removed the long-context cliff: absorbed attention scores all 32 heads against one shared 576-wide latent row and had fallen to 684 tok/s at 64K, so prefill chunks expand each latent segment to per-head K (192) / V (128) and run range attention. PR #50 stages that K/V in BF16, attends on 64-key tiles with ldmatrix, and packs three segments per scratch buffer. 2K–64K card sweep, #48 → #50: mean prefill 1,106 → 1,736 tok/s (+57%), 684 → 1,423 tok/s at 65,536 tokens (+108%), 1,670 → 2,048 at 8,192; mean decode 24.05 → 24.53. Versus the merged #49 path the same sweep is 1,597 → 1,736 tok/s (+8.7%) and 1,231 → 1,423 at 64K (+16%). Cold 65,536-token prefill after #49: 1,048 → 1,742 tok/s. Same argmax at every frontier. Kill switches: DS4_LING3VL_NO_MLA_EXPAND=1, DS4_LING3VL_MLA_KV_F32=1, DS4_LING3VL_NO_MLA_HMMA=1, DS4_LING3VL_PREFILL_CHUNK=2048, DS4_LING3VL_NO_BF16_REUSE=1, DS4_LING3VL_NO_BF16_VEC=1, DS4_MOTIF3_ATTN_HG_FILL=1. Campaign, long-context rounds and tests · Raw sweep CSVs, receipt and plot script.
!Ling-3.0-flash-VL MQ-Q5 2K–64K throughput, PR #48 vs #49 vs #50
Measured on this artifact. Curves are per-frontier medians; bands are the observed min–max over two #48 runs and three runs each of #49 and #50. One warm session per fresh process, 2,048-token incremental prefill and 128 greedy tokens per frontier, same artifact and prompt, SM 2177–2190 MHz while busy; no MTP (the family has none).
Support my work
I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.
While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.
Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.
<a href="https://www.buymeacoffee.com/baekpica" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee" style="height: 60px !important;width: 217px !important;"></a> <a href="https://github.com/sponsors/Baekpica" target="_blank"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="Sponsor Baekpica on GitHub" style="height: 60px !important;width: 217px !important;"></a>
This is a role-aware mixed-precision conversion of inclusionAI/Ling-3.0-flash-VL, pinned to revision 554184d95863a873051e8d8ccdb08c32c2c53a03.
These artifacts target ds4-dfm-rs, a Rust-host continuation of DwarfStar for explicit, hardware-validated large-model inference on NVIDIA DGX Spark. It preserves the antirez → Entrpi → Baekpica lineage and integrates a growing set of model families with minimal abstraction while retaining the native CUDA/MMQ kernels.
Why this model needs an asymmetric layout
BailingMoeV3VL is not a uniform transformer, and quantizing it as one wastes bits in the wrong places.
| Structure | Detail |
|---|---|
| Blocks | 42, in 7 groups of 6 |
| Attention | hybrid: 35 KDA linear-attention blocks + 7 MLA full-attention blocks (blk.5, 11, 17, 23, 29, 35, 41) |
| MoE | 512 routed experts, top-8, 8 groups / top-4 groups, sigmoid scoring, 1 shared expert, first_k_dense_replace = 2 |
| Expert width | moe_intermediate_size = 768 against hidden_size = 2560 |
| Vision | 27-block Qwen3-VL-style ViT, hidden = 1152, 2×2 spatial merge, disable_merger_proj |
| Positions | M-RoPE; mrope_section = [8, 12, 12] splits the 32 rotary pairs contiguously, and each pair is adjacent (2j, 2j+1), not half-offset; rope_theta = 6e6; partial_rotary_factor = 0.5 is the 64-dim rotary tail |
| Long context | Native 131,072; YaRN factor 2 to 262,144 (rope_type=yarn, original_max_position_embeddings=131072) |
Two consequences drive the recipe.
The KDA path is recurrent, so its error compounds. 35 of 42 blocks carry a delta-rule recurrent state whose update is governed by ssm_f_a (forget/decay), ssm_g_a (head-wise gate), ssm_beta, ssm_a (A_log) and ssm_dt. A per-token error in a decay or gate term is not a one-shot perturbation: it is multiplied into the state and carried forward for the rest of the sequence. These tensors are cheap — 105 tensors, 0.737B parameters, 1.3725 GiB at BF16 — so they are kept at BF16 and the control scalars stay F32.
The routed experts are effectively the entire parameter count. 120.80B of 124.41B parameters (97.1%) sit in ffn_{gate,up,down}_exps. Everything else in the language tower — both attention families, the shared expert, the dense FFN, the router, all norms — is 3.61B parameters and fits in BF16/F32 for 5.34 GiB. So the layout keeps all of it exact and spends the entire remaining budget on 120.80B expert parameters at an average of 5.02 bits.
Variant and download
| Variant | Routed expert tiers | Language payload | Vision payload | Total residency | Runtime status |
|---|---|---:|---:|---:|---|
| MQ-Q5-KDA-VIT-BF16 | Q5_K / Q4_K | 77.5054 GiB | 0.8179 GiB | 78.3233 GiB | structural audit passed; DGX Spark qualified |
ling_repo=Baekpica/Ling-3.0-flash-VL-Mixed-Quant-GGUF
ling_root=./Ling-3.0-flash-VL-Mixed-Quant-GGUF
ling_variant=MQ-Q5-KDA-VIT-BF16
hf download "$ling_repo" --include "$ling_variant/*" --local-dir "$ling_root"
(cd "$ling_root/$ling_variant" && sha256sum -c SHA256SUMS)
Keep every shard of the variant in one directory and open shard 00001 to load. The vision projector ships inside the same directory; it is not a separate download.
Quantization targets
| Model region | Tensors | Parameters | Target | Payload | Why |
|---|---:|---:|---|---:|---|
| Routed expert gate/up, interior blocks 7–33 | 54 | 54,358,179,840 | Q4_K | 28.4766 GiB | bulk; two of three expert matrices, interior depth |
| Routed expert down, all blocks 2–41 | 40 | 40,265,318,400 | Q5_K | 25.7812 GiB | expert output projection — one tier above gate/up everywhere |
| Routed expert gate/up, edge blocks 2–6, 34–41 | 26 | 26,172,456,960 | Q5_K | 16.7578 GiB | entry blocks and the clamped-SwiGLU span |
| Attention projections, KDA q/k/v/o + MLA q/o/gate | 161 | 1,652,080,640 | BF16 | 3.0772 GiB | shared by every token; not worth quantizing at 1.3% of params |
| KDA decay / gate / beta (ssm_f_a, ssm_g_a, ssm_beta) | 105 | 736,870,400 | BF16 | 1.3725 GiB | recurrent state control; error compounds along the sequence |
| Logit head (output.weight) | 1 | 402,391,040 | BF16 | 0.7495 GiB | maps directly to logits |
| Shared expert gate/up/down | 120 | 235,929,600 | BF16 | 0.4395 GiB | active on every token of every MoE block |
| Token embeddings | 1 | 402,391,040 | Q8_0 | 0.3982 GiB | lookup only; Q8_0 is near-lossless here |
| Norms, router (ffn_gate_inp), exp_probs_b, ssm_a, ssm_dt, KDA conv1d | 382 | 54,539,744 | F32 | 0.2032 GiB | routing and control state; never quantized |
| Leading dense FFN, blocks 0–1 | 6 | 94,371,840 | BF16 | 0.1758 GiB | dense path before the MoE stack begins |
| MLA latent projections (kv_a_mqa, k_b, v_b) | 21 | 39,682,048 | BF16 | 0.0739 GiB | rank-512 latent KV; tiny and precision-critical |
| Vision tower + projector (separate file) | 334 | 434,159,232 | BF16 / F32 | 0.8179 GiB | ViT blocks BF16, norms/patch/pos F32 |
Type census of the language GGUF: 414 BF16, 382 F32, 66 Q5_K, 54 Q4_K, 1 Q8_0 across 917 tensors.
Edge-block selection
gate/up is promoted from Q4_K to Q5_K on 13 of the 40 MoE blocks:
- Blocks 2–6, the first five MoE blocks, whose output every later block consumes.
- Blocks 34–41, the span where upstream clamps the SwiGLU.
config.jsonsets a non-zeroshare_expert_swiglu_limit_listfrom block 34 and a non-zeroexpert_swiglu_limit_listfrom block 35. A clamp is direct evidence of activation outliers, and outliers are where a block-wise expert quantizer loses the most.
down is not tiered this way: it is Q5_K on every MoE block, because it projects the expert result back into the residual stream and is uniformly more sensitive than gate/up.
No importance matrix
This artifact is calibration_dependency: none, matching Qwen3.8-Flash-Next-Mixed-Quant-SSD-PLE-GGUF.
With 512 experts at top-8 routing, per-expert activation statistics are intrinsically sparse — published attempts on this checkpoint family report routed experts with zero observations after a full calibration pass, and partially observed experts carry noisy column statistics. Because the recipe already removes the sensitive path from the quantizer entirely, an imatrix would only reweight the routed experts, which is exactly where its statistics are least trustworthy. Dropping it keeps the conversion deterministic and independently reproducible from the pinned checkpoint, with no binary calibration artifact in the trust chain.
Memory metrics
| Metric | Value |
|---|---:|
| Language GGUF logical parameters | 124,414,211,552 |
| Language GGUF tensor count | 917 |
| Language payload | 83,220,797,312 bytes / 77.5054 GiB |
| Language effective BPW | 5.3512 |
| Routed-expert parameters | 120,795,955,200 |
| Routed-expert payload | 71.0156 GiB (5.0500 BPW) |
| Non-expert language parameters | 3,618,256,352 |
| Non-expert language payload | 6.4898 GiB (15.4071 BPW) |
| Vision tower parameters | 434,159,232 |
| Vision tower + projector file | 878,174,368 bytes / 0.8179 GiB |
| Complete multimodal residency | 84,098,971,680 bytes / 78.3233 GiB |
| Source BF16 GGUF | 248,944,047,776 bytes / 231.8477 GiB |
These are exact tensor payload calculations using GGUF block sizes. Final file size, CUDA allocations, page cache, KV/recurrent state, and runtime workspace are separate.
Published artifact layout
MQ-Q5-KDA-VIT-BF16/
├── Ling-3.0-flash-VL-MQ-Q5-KDA-VIT-BF16-*.gguf
├── mmproj-Ling-3.0-flash-VL-BF16.gguf
├── SHA256SUMS
├── quant-recipe.yaml
├── tensor-types.txt
├── expected-inventory.json
├── artifact-manifest.json
├── verify-main-gguf.json
└── chat_template.jinja
tensor-types.txt is the exact --tensor-type-file consumed by llama-quantize; expected-inventory.json is the per-tensor target map that verify-main-gguf.json was produced against. chat_template.jinja is the base model's Bailing V3 template, copied byte for byte; the GGUF already carries it in tokenizer.chat_template, so it is published for toolchains that read the file instead.
| File | Bytes | SHA-256 |
|---|---:|---|
| Ling-3.0-flash-VL-MQ-Q5-KDA-VIT-BF16-00001-of-00003.gguf | 29,750,367,328 | 6902566540c98127… |
| Ling-3.0-flash-VL-MQ-Q5-KDA-VIT-BF16-00002-of-00003.gguf | 29,984,497,152 | 5c989b2ebe7af8f1… |
| Ling-3.0-flash-VL-MQ-Q5-KDA-VIT-BF16-00003-of-00003.gguf | 23,492,477,536 | 20f983e7c3fa888f… |
| mmproj-Ling-3.0-flash-VL-BF16.gguf | 878,174,368 | 2a8033028450df21… |
Published files total 84,105,516,384 bytes / 78.3294 GiB; the difference from the 83,220,797,312-byte tensor payload is GGUF headers, metadata and alignment.
Vision tower
The checkpoint sets disable_merger_proj, so model.visual.merger keeps only its norm and the real projection into the language model lives at the top level of the state dict as linear_proj.0 (4608 → 2560) and linear_proj.2 (2560 → 2560). The converter folds those into the merger namespace so they land as mm.0 / mm.2 in the projector file. vision_config.out_hidden_size is stale on this checkpoint (it reads 4096); the emitted clip.vision.projection_dim is the real 2560.
temporal_patch_size = 2 is handled by splitting the Conv3D patch embedding into v.patch_embd.weight and v.patch_embd.weight.1, following the Qwen3-VL convention.
Serve with ds4-dfm-rs
Build ds4-dfm-rs with make cuda-spark and point it at shard 00001 plus the projector beside it; runtime support below has the full command.
The language GGUF carries the VL metadata the runtime needs beyond the plain bailingmoe3 key set:
| Key | Value |
|---|---|
| bailingmoe3.rope.dimension_sections | [8, 12, 12, 0] |
| bailingmoe3.vision.image_token_id | 157157 |
| bailingmoe3.vision.video_token_id | 156909 |
| bailingmoe3.vision.start_token_id | 157158 |
| bailingmoe3.vision.end_token_id | 157159 |
For text-only positions M-RoPE assigns the same index to all three sections, so it reduces exactly to 1-D RoPE; the sections matter once image or video spans are laid out.
Runtime support
ds4-dfm-rs PR #46 adds this
family; PR #48 through
PR #50 are the CUDA
prefill/decode campaigns above. The YaRN factor-2 (262,144-token) branch,
PR #47, was closed without
merging: main serves this family up to its native 131,072 tokens and
refuses larger -c. One DGX Spark, CUDA, 128 GB unified memory:
LING=/path/to/MQ-Q5-KDA-VIT-BF16
KV=./ling-kv && mkdir -p "$KV"
./ds4-server --cuda \
-m "$LING/Ling-3.0-flash-VL-MQ-Q5-KDA-VIT-BF16-00001-of-00003.gguf" \
--vision "$LING/mmproj-Ling-3.0-flash-VL-BF16.gguf" \
--model-id Ling-3.0-flash-VL \
-c 65536 --max-seqs 2 --prefix-reuse partial \
--kv-disk-dir "$KV" --kv-disk-space 8G --kv-cache-min-tokens 64 \
--host 127.0.0.1 --port 8000 --no-update-check
-c 65536 is the qualified two-bank context; the runtime accepts up to the
native 131,072 on the factor-1 M-RoPE table.
| Surface | Ling-3.0-flash-VL |
|---|---|
| chat input | the GGUF's own Bailing V3 Jinja template; the legacy token builder refuses this family rather than approximating it |
| tools / thinking | GLM-style <tool_call> / <arg_key> / <arg_value> XML and <think> / </think> |
| still images | PNG/JPEG data URIs on OpenAI Chat Completions, on the serial lane beside the text banks |
| --max-seqs | persistent banks, no opt-in switch |
| prefix reuse | exact, fork, and partial fork from the recurrent checkpoint pool |
| --kv-disk-dir | session and per-bank payloads under the LNG3 layout magic |
| -c | 131,072 native (factor-1 M-RoPE); 65,536 qualified with two banks |
| MTP | none; bailingmoe3 has no predictor block |
Measured on one Spark. The two-bank HTTP numbers (PR #46) used -c 8192
at stock clocks. The ds4-bench rows are the PR #48–#50 campaigns on
the same artifact and machine, SM clocks observed at 2,177–2,197 MHz.
| Check | Result |
|---|---|
| 1,072-token cold prompt (HTTP, #46) | 1,461 tok/s prefill, 19.8 tok/s decode, TTFT 793 ms |
| second turn of the same conversation | 1,123 of 1,146 prompt tokens reused, TTFT 793 → 208 ms, reuse_kind=fork |
| still image, three colour bands | "Red, green, blue." — correct top to bottom, 0.7 s |
| still image, one shape | circle, blue — correct, 2.1 s |
| two banks | a short request returns in 0.9 s while a 256-token generation is still running |
| disk KV | LNG3 records written on bank evict and on shutdown; after a restart, 1,153 of 1,173 prompt tokens restored from disk into an empty bank |
| 8,192+64 ds4-bench, main 2faf925 (#46) | 1,141.97 tok/s prefill, 19.54 tok/s decode (median of 3) |
| 8,192+64 ds4-bench, main 8436382 (#48) | 1,888.77 / 1,893.34 tok/s prefill, 24.65 / 24.65 tok/s decode (two fresh processes) |
| 8,192+128 ds4-bench, 007f0e4 (#49) | 2,168 / 2,185 tok/s prefill, 25.4 / 25.7 tok/s decode (two fresh processes) |
| 2K–64K card sweep, #48 → #50 | mean prefill 1,106 → 1,736 tok/s, at 65,536 tokens 684 → 1,423 tok/s; mean decode 24.05 → 24.53 tok/s (medians of 2 / 3 run means). Versus #49: 1,597 → 1,736 (+8.7%), 1,231 → 1,423 at 64K |
A follow-up turn forks its prefix into a free bank, which preserves the source
conversation. When every bank already holds one, the fork is refused and the
turn prefills cold, so --max-seqs is the reuse budget as much as the
concurrency budget.
The runtime quotes 84.1 GiB of shared weights plus 1.37 GiB per 8,192-token
bank, a 2.39 GiB checkpoint pool for partial reuse, and an 870 MiB media
reserve. The recurrent state is 76.6 MiB for the whole model and does not grow
with context; only the 7 MLA blocks hold a per-token cache, at 8,064 bytes per
token — 504 MiB at 65,536 tokens, about 1 GiB at the full 131,072.
What this does not establish. No quality benchmark, no video input, no
HTTP serving above 8,192 context, no prompt above 65,536 tokens, no
131,072-token session, and no Metal, ROCm, or CPU path.
Reproduction
The upstream converter supports BailingMoeV3ForCausalLM (text-only Ling-3.0) but not the BailingMoeV3VLForConditionalGeneration wrapper or its vision tower. reproduction/conversion/bailingmoe3vl.py adds both: a text class that hoists text_config, supplies the num_shared_experts the VL config omits, drops the vision tensors and writes the M-RoPE/vision-token metadata; and a projector class that reuses the Qwen3-VL vision mapping with the linear_proj fold described above.
# 1. patched llama.cpp converter
cp reproduction/conversion/bailingmoe3vl.py llama.cpp/conversion/
python3 reproduction/scripts/register_converter.py --llama-cpp llama.cpp
# 2. BF16 language tower and BF16 projector
python3 llama.cpp/convert_hf_to_gguf.py "$src" --outtype bf16 \
--outfile Ling-3.0-flash-VL-BF16.gguf
python3 llama.cpp/convert_hf_to_gguf.py "$src" --mmproj --outtype bf16 \
--outfile mmproj-Ling-3.0-flash-VL-BF16.gguf
# 3. recipe, then mixed quantization
python3 reproduction/scripts/build_recipe.py \
--bf16 Ling-3.0-flash-VL-BF16.gguf --out-dir recipe
llama.cpp/build/bin/llama-quantize \
--tensor-type-file recipe/tensor-types.txt \
--token-embedding-type q8_0 --output-tensor-type bf16 \
Ling-3.0-flash-VL-BF16.gguf Ling-3.0-flash-VL-MQ-Q5-KDA-VIT-BF16.gguf Q5_K_M
# 4. split and audit
llama.cpp/build/bin/llama-gguf-split --split-max-size 30G \
Ling-3.0-flash-VL-MQ-Q5-KDA-VIT-BF16.gguf Ling-3.0-flash-VL-MQ-Q5-KDA-VIT-BF16
python3 reproduction/scripts/verify_gguf_structure.py *.gguf \
--expected recipe/expected-inventory.json --bf16 Ling-3.0-flash-VL-BF16.gguf \
--variant MQ-Q5-KDA-VIT-BF16 --out verify-main-gguf.json
build_recipe.py reads nothing but the BF16 GGUF, so the emitted tensor-types.txt and expected-inventory.json are byte-reproducible from the pinned checkpoint.
Verification
Reported in verify-main-gguf.json:
- tensor set identical to the expected inventory, no missing / extra / renamed tensors
- every tensor shape equal to the BF16 source
- every tensor on its recipe target type, with the payload size that type implies
- aggregate payload equal to the recipe to the byte
- every
BF16andF32tensor byte-identical to the BF16 source, compared by SHA-256 per tensor - per-file SHA-256 in
SHA256SUMS
Result:
| Check | Outcome |
|---|---|
| Tensor set vs expected inventory | 917 / 917, no missing, extra or renamed |
| Per-tensor target type | all on target: {'BF16': 414, 'F32': 382, 'Q4_K': 54, 'Q5_K': 66, 'Q8_0': 1} |
| Aggregate payload | 83,220,797,312 bytes, exact |
| Effective BPW | 5.3512 |
| BF16/F32 tensors byte-identical to BF16 source | 796 compared, 0 failed |
| Errors | 0 |
verify-main-gguf.json also records 105 informational notes: llama-gguf-split drops a trailing 1-dim from the 105 KDA ssm_conv1d_* tensors ([4, 1, 4096, 1] → [4, 1, 4096]). GGUF stores n_dims explicitly and the payload is unchanged, so this is recorded rather than treated as a defect.
Functional smoke check: the split artifact loads on 2×H200 with -ngl 99, and greedy decoding produces coherent, correct output (arithmetic and a technical explanation) at ~111 tok/s generation, ~127 tok/s prompt. This is a load-and-decode check, not a quality benchmark.
Independent corroboration of the conversion itself: the tensor set emitted here (917 tensors, of which 382 stay F32) matches the counts reported by an unrelated third-party conversion of the same checkpoint.
Known limitations
- Two-bank HTTP qualification was at 8,192 context on one DGX Spark.
ds4-benchthroughput is measured through 65,536 tokens in one warm session (2K–64K sweep above). Longer prompts, the full 131,072-token context, video input, and every non-CUDA backend are outside what was measured; see runtime support. - No quality benchmark is published here. No KLD, perplexity, or vision benchmark was measured for this variant.
- Upstream llama.cpp cannot load the VL pair as shipped. It has a
bailingmoe3text graph, but noBailingMoeV3VLconverter registration and no vision bridge for this projector. The converter patch underreproduction/is required to regenerate the artifacts, and a runtime bridge is required to serve images. - The M-RoPE section metadata is carried for the runtime; it is not consumed by the upstream
bailingmoe3graph, which applies 1-D RoPE.
License
MIT, inherited from inclusionAI/Ling-3.0-flash-VL.
Run Baekpica/Ling-3.0-flash-VL-Mixed-Quant-GGUF with guIDE
Download guIDE — the AI-native code editor with local LLM inference and 69 built-in tools.
Source: Hugging Face · Compare models