steampunque/GLM-4.6V-Flash-MP-GGUF overview

---

license: mit

base_model: zai-org/GLM-4.6V-Flash

base_model_relation: quantized

tags:

• GLM 4.6V Flash
• GGUF
• quantized
• 4-bit

---

Mixed Precision GGUF layer quantization of GLM-4.6V-Flash by zai-org

Original model: https://huggingface.co/zai-org/GLM-4.6V-Flash

The hybrid quant employs different quantization levels on a per layer basis to enable

both high performance and small file size at the same time. An extended layer definition E quant Q4_E_H for the model is

defined as follows (updated 10/17/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_666"], [1 ,"Q4_K_666"], [2 ,"Q4_K_655"], [3 ,"Q4_K_644"], [4 ,"Q4_K_544"], [5 ,"Q4_K_544"], [6 ,"Q4_K_544"], [7 ,"Q4_K_544"],
   [8 ,"Q4_K_544"], [9 ,"Q4_K_544"], [10,"Q4_K_544"], [11,"Q4_K_544"], [12,"Q4_K_544"], [13,"Q4_K_544"], [14,"Q4_K_544"], [15,"Q4_K_544"],
   [16,"Q4_K_544"], [17,"Q4_K_544"], [18,"Q4_K_544"], [19,"Q4_K_544"], [20,"Q4_K_544"], [21,"Q4_K_544"], [22,"Q4_K_544"], [23,"Q4_K_544"],
   [24,"Q4_K_544"], [25,"Q4_K_544"], [26,"Q4_K_544"], [27,"Q4_K_544"], [28,"Q4_K_544"], [29,"Q4_K_544"], [30,"Q4_K_544"], [31,"Q4_K_544"],
   [32,"Q4_K_544"], [33,"Q4_K_544"], [34,"Q4_K_544"], [35,"Q4_K_544"], [36,"Q4_K_544"], [37,"Q4_K_544"], [38,"Q4_K_544"], [39,"Q4_K_644"],
   [40,"Q4_K_644"], [41,"Q4_K_654"], [42,"Q4_K_655"], [43,"Q4_K_665"], [44,"Q4_K_666"], [45,"Q5_K_668"], [46,"Q6_K_866"]
   ]'
   FLAGS="--token-embedding-type Q5_K --output-tensor-type Q6_K --layer-types-high --tensor-pad [[13696,13824],[27392,27648,2]] --override-kv glm4.feed_forward_length=int:13824"

Model FFN length is padded from 13696 to 13824 to allow use of K quants in the layers instead of fallback legacy quants. Minimum quant

across layers is Q4_K to minimize entropy loss. The bpw is sized to enable the model + vision tower to run on an 8G VRAM card and then

evaluated for acceptable vision and reasoning performance across a curated set of test prompts.

Comparison:

Quant | size | PPL | Comment

---------|---------|------|-----------

Q4_K_S | 5.8e9 | 12.3 | -

Q4_E_H | 5.6e9 | 12.1 | Hybrid quant with Q5_K embedding Q6_K output

Usage:

This is a RL trained (thinking) vision model. The layer quants for this model were evaluated on a set of test/eval prompts using

greedy sampling. The model appears to be robust against infinite generations on the eval prompts with greedy sampling,

reliably converging to an answer on all tested prompts. Image mode was tested on a small set of images and found to be both functional and accurate.

This model will respond in Chinese with no system prompt. For english responses, the following system prompt can be used:

SYSTEM="language = en"

In tests this system prompt will cause the model to both reason and respond in english. Note this system prompt is completely

empirical and it appears necessary to change it when the model quant changes to make it robustly respond in english. The

prompt above is for Q4_E_H and may or may not work well with other quants.

The model can be speculated with Qwen3 0.6B if the inference platform can handle vocab translation between draft and target.

Approx performance using a downstream speculator with llama.cpp on one 4070 (12G VRAM) GPU with fixed spec block length ND:

ND | QKV | NKV | gen tps | Comment

---|-----|-----|---------|--------

3 | F16 | 38k | 108 | llama.cpp b11716 downstream spec

0 | F16 | 128k| 75 | ""

3 | Q8_0| 42k | 107 | ""

0 | Q8_0| 128k| 74 | ""

Vision:

The model went 2 for 2 on a couple tough bird ID images.

Benchmarks:

Vision benchmarks for the model are given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

| Link | Type | Size/e9 B | Notes |

|------|------|-----------|-------|

| GLM-4.6V-Flash.Q4_E_H.gguf | Q4_E_H | 5.6e9 B | ~0.2B smaller than Q4_K_S |

| GLM-4.6V-Flash.mmproj.gguf | F16 | 1.8e9 B | multimedia projector |

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040