---

license: apache-2.0

language:

- en

- zh

library_name: llama.cpp

pipeline_tag: text-generation

tags:

- qwen3.5

- moe

- gguf

- imatrix

- lynnstyle

- multimodal

- no-mtp

---

Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF

!44G LynnStyle GGUF overview

> LCB note: LCB is the standard abbreviation for LiveCodeBench. It selects competitive-programming problems published after the evaluated model's training-data cutoff date and judges code correctness by executing test cases, helping reduce training-data contamination. The LCB100 reported on this card is a fixed 100-problem coding evaluation under the same protocol.

This repository contains the 44G LynnStyle layer-wise mixed-precision GGUF of Qwen3.5-122B-A10B. The main GGUF already includes this training branch's step64 LoRA update, so no separate LoRA adapter is required. An optional multimodal mmproj is included.

The repository does not ship an MTP sidecar for two practical reasons: the main model, KV cache, and runtime overhead need to fit reliably on one 48GB GPU, and controlled testing found that quantizing the MTP sidecar reduced output quality. All reported scores and recommended commands therefore use NO MTP.

> At a glance

>

> - Main GGUF: 44.61 GB (41.55 GiB); main model plus mmproj: 45.23 GB (42.13 GiB).

> - This 44G mixed-precision scheme keeps critical pathways at high precision while letting the largest ordinary expert layers absorb most of the compression.

> - Put plainly: components that strongly affect understanding, reasoning, and output convergence are compressed conservatively; the numerous, space-heavy ordinary routed-expert weights carry the aggressive compression.

> - Final NO-MTP evaluation: MMLU 91.6% (458/500), GPQA 86.4% (171/198), and LCB 82.0% (82/100).

> - Verified runtime: C4 = 4 × 32K, 131072 total context, Q4 K/V cache, full GPU offload, --fit off; peak VRAM was 45,551 MiB.

Repository files

| File | Role | Size | SHA256 |

|---|---|---:|---|

| Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.gguf | Main model with the step64 LoRA update merged | 44,612,547,968 bytes | 6a25dfc683de6754ffd632141a9a454f5f1daa0ed34fe62c5192725fdb0b4910 |

| Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF-mmproj-Q8_0.gguf | Optional vision projector | 619,212,000 bytes | e77a1be433e92791e9ee85f99963dad6421aa8188314d9a6670bb9ab196034eb |

| Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.SHA256SUMS | File-integrity checks | — | — |

| Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.manifest.json | Build-time manifest | — | — |

The mmproj filename keeps the Q8_0 tier label, while its actual conversion contains Q8_0 weights with F16 fallbacks and small F32 structures. The main model can be used on its own for text inference; add --mmproj only when processing visual input.

Training and release lineage

• Architecture: qwen3next; the main GGUF contains 807 tensors.
• GPT5.6Sol-SFT in the repository name identifies this model's training branch.
• The release-build audit confirms that the step64 LoRA was merged into the quantization input. The final main GGUF does not require --lora.
• The 44G artifact uses imatrix-assisted LynnStyle layer-wise mixed precision rather than assigning one quantization type to all 807 tensors.
• The public quality results were produced without MTP, and the recommended runtime must not add an external --model-draft.

Quantization design

Actual 807-tensor distribution

| GGUF type | Count |

|---|---:|

| F32 | 313 |

| IQ2_XS | 126 |

| Q5_K | 20 |

| Q6_K | 108 |

| Q8_0 | 240 |

| Total | 807 |

Protection map

| Protection group | Layers / tensor family | Actual type | Count | Purpose |

|---|---|---:|---:|---|

| Input/output anchors | token_embd.weight, output.weight | Q5_K | 2 | Preserve vocabulary entry and final token selection |

| Shared-expert main matrices | All ffn_*_shexp.weight tensors | Q8_0 | 192 | Preserve the shared path traversed by every token |

| Full attention | Every present attn_q/k/v/output.weight tensor | Q8_0 | 48 | Preserve full-attention reads, writes, and long-range integration |

| SSM main matrices | All ssm_ba/in/out.weight tensors | Q6_K | 108 | Preserve the linear-attention/state-space pathway |

| Routed experts: high protection | Gate/up/down on L26 and L43–L47 | Q5_K | 18 | Protect mid-to-late hot layers and output convergence |

| Routed experts: compression group | Gate/up/down on the remaining 42 layers | IQ2_XS | 126 | Carry most of the size reduction |

| Router, norms, biases, and other small structures | Remaining structures | F32 | 313 | Avoid low-bit damage to small, high-leverage controls |

The protection unit is layer × path × tensor family. Standard GGUF stores the routed experts of one layer in a stacked tensor, so this build protects entire layer-level expert tensors rather than assigning a separate precision to each expert inside one tensor.

REAP/router observations identify task pathways, while the imatrix constrains quantization error. Neither replaces final evaluation. L26 and L43–L47 were preregistered as the high-protection set during the build; this does not claim that any single layer independently determines model quality. Release acceptance comes from the complete MMLU500, GPQA198, LCB100, and real-load gates.

What “mixed precision” means here

• Weights: shared-expert and attention weights remain Q8; SSM weights remain Q6; routed experts on L26 and L43–L47 remain Q5; the other routed-expert layers use IQ2_XS.
• Runtime compute: llama.cpp still uses FP16/FP32 compute and accumulation paths. Activations are not uniformly quantized to INT4.
• K/V cache: the verified C4 runtime uses a q4_0 K/V cache. This is runtime context storage and does not change the model-file weights.
• Compact technical description: tensor-wise mixed weights at approximately 2.85 serialized bits per original BF16 weight, FP16/FP32 activation compute, and a validated Q4 K/V cache.

The main GGUF is 65.65% smaller than the 129,871,496,512-byte original Q8 and 82.17% smaller than the 250,196,479,366-byte BF16 source.

Q8 comparison and full evaluation

The fixed evaluation set contains MMLU500, GPQA198, and LCB100. Generation used temperature 0.6, top_p=0.95, and at most 32768 output tokens per slot. The 44G evaluation used C4, 32K per slot, Q4 K/V cache, and NO MTP; MMLU was 5-shot.

| Version | MMLU500 | GPQA198 | LCB100 | Notes |

|---|---:|---:|---:|---|

| Original Q8 baseline | 93.4% (467/500) | 89.4% (177/198) | 78.0% (78/100) | Q8 GPQA was rejudged from the saved 198 rows using the final standalone answer; Q8 LCB is natural 32K |

| 44G, natural 32K | 91.6% (458/500) | 85.4% (169/198) | 81.0% (81/100) | GPQA had 2 length IDs; LCB had 10 length IDs |

| 44G, final corrected gate | 91.6% (458/500) | 86.4% (171/198) | 82.0% (82/100) | Each natural-32K length ID was independently retested once at 36K and replaced by its actual result |

| 44G final vs. Q8 | -1.8 pp | -3.0 pp | +4.0 pp* | This is not a quantization-only ablation |

\* The Q8 LCB baseline did not receive a symmetric 36K length-only retest. Under the same natural-32K condition, 44G scored 81.0% (81/100) versus Q8's 78.0% (78/100), a +3.0 pp difference.

This comparison is not a pure “change only the quantization bits” A/B test. The Q8 artifact is the original quality baseline, while the 44G main model also includes the step64 LoRA update. The measured differences therefore combine the training update and mixed quantization.

Parsing and length-only retest policy

• An older Q8 GPQA summary incorrectly reported 50.0% (99/198) because it prepended reasoning_content to the final answer before parsing. Rejudging the same saved 198 rows by the final standalone A/B/C/D choice gives 89.4% (177/198); all 198 rows ended with stop, and parse/error/timeout counts were zero.
• Natural-32K 44G GPQA scored 85.4% (169/198). Its two length IDs each received one independent 36K retest; both ended with stop and were correct, producing 86.4% (171/198).
• Natural-32K 44G LCB scored 81.0% (81/100). Its ten length IDs each received one 36K retest; one became a passing solution, producing 82.0% (82/100). After that single retest, six remained length-limited and four remained empty; API errors and request timeouts were zero.
• LCB wrong answers, runtime errors, and test-case timeouts count as incorrect. They are never silently promoted to passes.

The final corrected gate passed. Peak VRAM was 45,551 MiB, below the 49,152 MiB limit.

Download and verify

from huggingface_hub import snapshot_download

model_dir = snapshot_download(
    repo_id="nerkyor/Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF"
)
print(model_dir)

Or:

git lfs install
git clone https://huggingface.co/nerkyor/Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF
cd Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF
sha256sum -c Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.SHA256SUMS

llama.cpp

Use a recent llama.cpp build with qwen3next and the corresponding multimodal projector support.

Text, one slot

llama-server \
  -m Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.gguf \
  -ngl all --fit off \
  -c 32768 -np 1 -n 32768 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 0.0.0.0 --port 8080 \
  --metrics --no-webui --cache-ram 0

Verified C4 with the optional vision projector

llama-server \
  -m Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.gguf \
  --mmproj Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF-mmproj-Q8_0.gguf \
  -ngl all --fit off \
  -c 131072 -np 4 -n 32768 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 0.0.0.0 --port 8080 \
  --metrics --no-webui --cache-ram 0

Do not add --model-draft: this repository does not publish MTP, and its quality scores were not produced with MTP.

Memory guidance and limitations

• A 48 GiB GPU is a close fit. C4 peaked at 45,551 MiB in the verified configuration, but display usage, driver reservations, llama.cpp revisions, and larger K/V caches can reduce headroom. Reduce concurrency or context first if VRAM is tight.
• The mmproj passed GGUF structure and file-integrity audits, but this card does not claim an independent vision-quality benchmark. Do not interpret the text scores as vision scores.
• The protected shared-expert, attention, and SSM pathways preserve strong quality, but a small number of long coding problems still fail to converge within 36K. Execute tests and independently verify generated code, scientific claims, and high-stakes outputs.
• The manifest captures the binary-upload-stage snapshot. The complete scores in this card come from the later frozen NO-MTP corrected gate.

Links

• Official Qwen3.5-122B-A10B model
• ModelScope mirror and Chinese-first card
• LynnMerkyor/Lynn project

---

中文

!44G LynnStyle GGUF 中文总览

> LCB 说明： LCB 是 LiveCodeBench 的标准缩写。它按照被测模型训练数据的截止日期，选取该日期之后发布的竞赛编程题，并通过执行测试用例判定代码正确性，以降低训练数据污染。本卡的 LCB100 为固定 100 题同口径编码测试。

Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF

这是 Qwen3.5-122B-A10B 的 44G LynnStyle MoE 分层混合精度 GGUF。主模型已合入本训练分支的 step64 LoRA 更新，下载后不需要再挂载独立 LoRA；仓库另附可选的视觉 mmproj。为了让主模型、运行时缓存和必要开销稳定装进单张 48GB 显卡，本仓不附带 MTP sidecar；同时，受控测试发现量化后的 MTP 会降低输出质量，因此公开质量分数与推荐启动方式全部采用 NO MTP。

> 先看结论

>

> - 主 GGUF：44.61 GB（41.55 GiB）；主模型加 mmproj：45.23 GB（42.13 GiB）。

> - 这份44G的混合精度方案：关键通路保留高精度，体积最大的普通专家层承担主要压缩。

> - 换句话说，容易影响理解、推理和输出收口的部分尽量少压；数量最多、最占空间的普通专家权重压得更狠。模型推理本身仍使用 FP16 / FP32，Q4 只用于运行时的上下文缓存。

> - NO-MTP 全量终局：MMLU 91.6%（458/500）、GPQA 86.4%（171/198）、LCB 82.0%（82/100）。

> - 已验证 C4 = 4 × 32K、总上下文 131072、Q4 KV、全 GPU、--fit off，峰值 45,551 MiB。

> - ModelScope 侧栏可能把该文件标成 unknown / IMATRIX；真实类型以本卡、manifest 和 GGUF 解析为准。

仓库文件

| 文件 | 作用 | 大小 | SHA256 |

|---|---|---:|---|

| Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.gguf | 主模型，已合入 step64 LoRA | 44,612,547,968 bytes | 6a25dfc683de6754ffd632141a9a454f5f1daa0ed34fe62c5192725fdb0b4910 |

| Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF-mmproj-Q8_0.gguf | 可选视觉投影 | 619,212,000 bytes | e77a1be433e92791e9ee85f99963dad6421aa8188314d9a6670bb9ab196034eb |

| Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.SHA256SUMS | 文件校验 | — | — |

| Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.manifest.json | 构建时 manifest | — | — |

mmproj 文件名沿用 Q8_0 档名；实际转换结果为 Q8_0 + F16 fallback + F32 小结构，不是所有 tensor 都统一 Q8。主模型可独立用于文本推理；视觉输入时再添加 --mmproj。

训练与发布谱系

• 架构：qwen3next，主 GGUF 共 807 tensors。
• 仓库命名中的 GPT5.6Sol-SFT 对应该模型训练分支。
• 发布构建审计确认：step64 LoRA 已先合入量化输入，最终主 GGUF 不需要 --lora。
• 44G 文件使用带 imatrix 的 LynnStyle 分层混合精度，而不是把 807 个 tensor 一刀切成单一量化类型。
• 本仓没有 MTP：一是为了给单张 48GB 显卡上的主模型、KV cache 和运行时开销留出空间；二是受控测试观察到量化后的 MTP 会降低输出质量。因此公开全量采用 NO MTP，启动时不要添加外部 --model-draft。

量化保护方案

实际 807-tensor 分布

| GGUF 类型 | 数量 |

|---|---:|

| F32 | 313 |

| IQ2_XS | 126 |

| Q5_K | 20 |

| Q6_K | 108 |

| Q8_0 | 240 |

| 合计 | 807 |

真实保护表

| 保护桶 | 层 / tensor family | 实际类型 | 数量 | 目的 |

|---|---|---:|---:|---|

| 输入输出锚点 | token_embd.weight、output.weight | Q5_K | 2 | 保护词表进入与最终输出 |

| shared-expert 主矩阵 | 全层 ffn_*_shexp.weight | Q8_0 | 192 | 保护每个 token 都会经过的共享路径 |

| full-attention | 全部实际存在的 attn_q/k/v/output.weight | Q8_0 | 48 | 保护全注意力读写与长程整合 |

| SSM 主矩阵 | 全部 ssm_ba/in/out.weight | Q6_K | 108 | 保护线性注意力 / 状态空间主通路 |

| routed experts：高保护 | L26、L43–L47 的 gate/up/down | Q5_K | 18 | 保护中后段热点与输出收口 |

| routed experts：压缩桶 | 其余 42 层的 gate/up/down | IQ2_XS | 126 | 承担主要体积压缩 |

| 路由、norm、bias 等小结构 | 其余未降档结构 | F32 | 313 | 避免小而高杠杆的控制结构被低比特化 |

这里的保护单位是 层 × 路径 × tensor family。标准 GGUF 会把同层 routed experts 堆叠到同一张大 tensor 中，因此本档保护的是整层专家张量，而不是在一张 tensor 内给单个 expert 指定不同精度。

REAP / 路由观测用于识别任务路径，imatrix 用于约束量化误差；二者都不能替代最终测评。L26、L43–L47 是构建时预注册的高保护集合，并不等于声称某一层单独“决定”全部能力。发布结论只由完整 MMLU500、GPQA198、LCB100 和真实加载门禁给出。

44G 具体用了什么精度

• 权重怎么压：shared-expert 与 attention 保留 Q8，SSM 保留 Q6，L26 与 L43–L47 的 routed experts 保留 Q5；其余 routed experts 使用 IQ2_XS 承担主要体积压缩。
• 推理怎么算：llama.cpp 运行时仍使用 FP16 / FP32 路径与累加，没有把激活统一压成 INT4。
• KV cache 怎么放：公开 C4 门禁使用 q4_0 K/V cache。它只影响运行时上下文缓存，不会改变模型文件里的权重。
• 如果需要写技术规格，可以记为 Mixed-W≈2.85bpw / A16–32 / KV4；面向普通用户直接称为 44G 分层混合精度 GGUF 更清楚。

主 GGUF 相比 129,871,496,512-byte 原始 Q8 缩小 65.65%，相比 250,196,479,366-byte BF16 缩小 82.17%。

Q8 对照与 44G 全量结果

主评测使用固定 MMLU500 / GPQA198 / LCB100 资产；生成温度 0.6、top_p=0.95、每路最大输出 32768 tokens。44G 使用 C4、每路 32K、Q4 KV、NO MTP；MMLU 为 5-shot。所有百分比均在前，绝对值附后。

| 版本 | MMLU500 | GPQA198 | LCB100 | 说明 |

|---|---:|---:|---:|---|

| 原始 Q8 基线 | 93.4%（467/500） | 89.4%（177/198） | 78.0%（78/100） | Q8 GPQA 对同一 198 条保存结果做尾部最终选项重判；LCB 为自然 32K |

| 44G，自然 32K | 91.6%（458/500） | 85.4%（169/198） | 81.0%（81/100） | GPQA 有 2 个 length ID；LCB 有 10 个 length ID |

| 44G，最终修正门禁 | 91.6%（458/500） | 86.4%（171/198） | 82.0%（82/100） | 只对自然 32K 的 length ID 各做一次独立 36K 补测并替换原题 |

| 44G 最终 vs Q8 | -1.8pp | -3.0pp | +4.0pp* | 不是纯量化单变量 A/B，见下方说明 |

\* Q8 LCB 没有做对称的 36K length-only 补测。完全同为自然 32K 时，44G 是 81.0%（81/100），相对 Q8 的 78.0%（78/100） 为 +3.0pp。

这张表不是“只改量化位宽”的纯 A/B：Q8 是原始质量基线，而 44G 主模型已经包含 step64 LoRA 更新。因此差值同时反映训练更新与混合量化，不能全部归因于量化。

解析与补测口径

• Q8 GPQA 的旧 summary 曾因把 reasoning_content 拼在最终答案前而错误统计为 50.0%（99/198）。同一份 198 条 JSONL 采用“最后一个独立 A/B/C/D”规则重判后为 89.4%（177/198）；198/198 均为 stop，parse/error/timeout 为 0。
• 44G GPQA 自然 32K 为 85.4%（169/198）；2 个 length ID 在一次 36K 补测中均 stop 且答对，最终为 86.4%（171/198）。
• 44G LCB 自然 32K 为 81.0%（81/100）；10 个 length ID 各补测一次 36K，1 题转为通过，最终为 82.0%（82/100）。修正后仍有 6 个 length、4 个 empty；API error 与请求 timeout 均为 0。
• LCB 的代码执行 wrong answer、runtime error 或测试用例超时均按错误计分，不会静默改判为正确。

最终 corrected gate 为 PASS；运行峰值 45,551 MiB，低于 49,152 MiB 门槛。

下载与校验

from huggingface_hub import snapshot_download

model_dir = snapshot_download(
    repo_id="nerkyor/Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF"
)
print(model_dir)

或：

git lfs install
git clone https://huggingface.co/nerkyor/Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF
cd Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF
sha256sum -c Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.SHA256SUMS

llama.cpp 启动

文本单槽

llama-server \
  -m Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.gguf \
  -ngl all --fit off \
  -c 32768 -np 1 -n 32768 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 0.0.0.0 --port 8080 \
  --metrics --no-webui --cache-ram 0

已验证的 C4 + 可选视觉投影

llama-server \
  -m Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF.gguf \
  --mmproj Qwen3.5-122B-A10B-44GB-GPT5.6Sol-SFT-LynnStyle-GGUF-mmproj-Q8_0.gguf \
  -ngl all --fit off \
  -c 131072 -np 4 -n 32768 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 0.0.0.0 --port 8080 \
  --metrics --no-webui --cache-ram 0

请使用支持 qwen3next 与相应 multimodal projector 的较新 llama.cpp。不要添加 --model-draft：本仓没有发布 MTP，质量分数也不是 MTP 成绩。

显存与使用限制

• 48GiB 档属于贴边运行：已验证 C4 峰值为 45,551 MiB，但桌面显示、驱动保留、不同 llama.cpp 版本和更大 KV cache 都会改变余量。显存紧张时先降并发或上下文。
• mmproj 已通过 GGUF 结构与文件审计；本卡没有发布独立视觉质量基准，不应把文本分数解释成视觉分数。
• 44G 保留了高质量 shared-expert、attention 与 SSM 主路径，但仍有少量长代码题在 36K 下不能收口。高风险代码、科研和事实任务仍应执行测试与外部核验。
• ModelScope 侧栏无法准确表达逐 tensor 混合精度；请按完整文件名下载，不要把 unknown / IMATRIX 当成损坏。
• 仓内 manifest 保存的是二进制上传阶段快照；本卡的完整全量数据来自随后完成并冻结的 NO-MTP corrected gate。

相关链接

• Qwen3.5-122B-A10B 官方模型
• 本 Hugging Face GGUF 仓
• ModelScope 镜像
• 配套工程：LynnMerkyor/Lynn