---

base_model: huihui-ai/Huihui-GLM-4.7-Flash-abliterated

base_model_relation: quantized

quantized_by: cognitivers

library_name: gguf

pipeline_tag: text-generation

license: mit

language:

- en

- zh

tags:

- gguf

- llama.cpp

- abliterated

- uncensored

- moe

- glm4

- imatrix

---

GLM-4.7-Flash-abliterated, 12 GB GGUF

GGUF quantizations of huihui-ai/Huihui-GLM-4.7-Flash-abliterated,

built for one specific target: a 12 GB consumer GPU with the routed experts offloaded to system RAM.

These are not another uniform ladder. The bit budget is allocated by the role each tensor plays at

inference time, which is where a Mixture-of-Experts model leaves a lot on the table.

Why a split-aware mix

GLM-4.7-Flash is a 29.94B MoE with roughly 3.3B active parameters per token. llama.cpp's

--cpu-moe / --n-cpu-moe moves exactly \.ffn_(up|down|gate|gate_up)_(ch|)exps to the CPU.

Everything else stays resident on the GPU forever:

| Block | Params | Share of model | Lives on |

|---|---|---|---|

| MLA attention (47 layers) | 1.02B | 3.3 % | GPU |

| Shared experts + dense layer 0 | 0.50B | 1.6 % | GPU |

| Token embeddings + output head | 0.63B | 2.0 % | GPU |

| Routed experts (64 x 46) | 28.2B | 90.4 % | RAM, or GPU via -ncmoe |

The GPU-resident part is only 2.15B parameters. Keeping it at q8_0 costs about 2.3 GB, which is

cheap on a 12 GB card. A uniform ladder spends the same bits per weight on those 2.15B as on the

28.2B of routed experts, and that is quality given away for nothing.

So: **q8_0 for the router, the MLA attention, the shared experts, the dense layer and the output

head; q6_k for the token embeddings; and the routed experts compressed hard.** The first four MoE

layers get one extra step, since early layers tolerate low-bit worse.

Files

| File | Size | Routed experts | Intended use |

|---|---|---|---|

| GLM-4.7-Flash-abliterated.Q4_K-SPLIT.gguf | 18.2 GB | q4_k (first 4 layers q5_k) | Recommended. 12 GB VRAM + ~16 GB RAM |

| GLM-4.7-Flash-abliterated.IQ2_M-SPLIT.gguf | 11.3 GB | iq2_s (first 4 layers iq3_xxs) | Fits entirely in 12 GB VRAM, at a measured reasoning cost |

| imatrix.gguf | 72 MB | | The importance matrix used, so the recipe is reproducible |

Both were quantized from a BF16 conversion of the original safetensors (never re-quantized from a

smaller GGUF), with an importance matrix computed on calibration_datav3 over 125 chunks.

How to run it on 12 GB

# Recommended: 26.3 tok/s, 10.19 GB VRAM, 1.68 GB headroom
llama-server -m GLM-4.7-Flash-abliterated.Q4_K-SPLIT.gguf -ngl 99 -ncmoe 24 -c 8192 -fa on

# Aggressive: 33.5 tok/s, but only 0.39 GB of VRAM left. Long prompts may OOM.
llama-server -m GLM-4.7-Flash-abliterated.Q4_K-SPLIT.gguf -ngl 99 -ncmoe 20 -c 8192 -fa on

# Minimum VRAM: all experts in RAM, 2.82 GB on the card, 16.4 tok/s
llama-server -m GLM-4.7-Flash-abliterated.Q4_K-SPLIT.gguf -ngl 99 -cmoe -c 8192 -fa on

-ncmoe N keeps the routed experts of the first N layers (out of 46) in system RAM. Lower N means

more experts on the GPU and more speed, until you run out of VRAM.

!12 GB drill

Measured on an RTX 4070 Ti (11874 MiB usable), context 8192, llama-bench with -p 512 -n 128 -r 2.

-ncmoe 18 and below abort with a CUDA allocation error on this card.

Quality

Everything below is measured against a BF16 conversion of the same checkpoint, not against a

smaller quantization, on wikitext-2-raw test (sha256 173c87a5..., 565 chunks, ctx 512).

| Model | Size | PPL ratio | Mean KLD | Median KLD | KLD p99 | Same top-token |

|---|---|---|---|---|---|---|

| Q8_0 (reference point) | 31.8 GB | 1.017 | 0.0298 | 0.0020 | 0.106 | 96.43 % |

| Q4_K-SPLIT | 18.2 GB | 1.006 | 0.0871 | 0.0088 | 0.633 | 92.63 % |

| IQ2_M-SPLIT | 11.3 GB | 1.166 | 0.2302 | 0.0723 | 2.993 | 82.35 % |

!Fidelity vs size

A PPL ratio near or below 1.0 is not evidence of being better than the reference. Low-bit

quantization can smooth a corpus and lower perplexity while still diverging from the original

distribution, which is exactly why top-token agreement and KLD are reported next to it.

Reasoning is not free below ~3 bits per expert

GSM8K, 300 problems, paired against the same baseline, bootstrap CI over paired per-item differences:

| Model | Accuracy drop | 95 % upper bound | Verdict at a 3 pp bar |

|---|---|---|---|

| Q4_K-SPLIT | 0.33 pp | 2.33 pp | passes |

| IQ2_M-SPLIT | 5.33 pp | 8.00 pp | fails |

Chain-of-thought length did not inflate (+1.8 %, upper bound +8.8 %) and tool-calling was unchanged

(call rate, JSON validity and argument correctness all 1.00 in both).

Be aware of what this means: the 11.3 GB file measurably costs you reasoning accuracy. It is

published because it is the best option we could measure at that size, not because it is lossless.

If you can spare the system RAM, use the 18.2 GB file with -ncmoe.

Head to head at the same size

The natural comparison is mradermacher/Huihui-GLM-4.7-Flash-abliterated-i1-GGUF, a

well-made imatrix ladder over the same abliterated checkpoint. Its i1-IQ3_XXS (11.65 GB) is

the closest neighbour in size to our 11.30 GB file, so we are competing 0.35 GB smaller.

Both were measured in the same session, against the same BF16 base logits, with the same llama.cpp

build and the same corpus. Numbers published elsewhere are not comparable to these; these are.

| | Ours, IQ2_M-SPLIT | i1-IQ3_XXS |

|---|---|---|

| Size | 11.30 GB | 11.65 GB |

| Mean KLD vs BF16 | 0.2302 | 0.3916 |

| Median KLD | 0.0723 | 0.1749 |

| KLD p99 | 2.993 | 4.201 |

| Same top-token | 82.35 % | 75.25 % |

| GSM8K drop | 5.33 pp | 9.33 pp |

!Head to head

Smaller file, 41 % lower mean KLD, 7.1 points more top-token agreement, and 43 % less damage to

reasoning. That is the case for allocating bits by role instead of uniformly.

The abliteration survives quantization

This matters more than perplexity for a model published as abliterated, and it is the one claim a

quantizer can easily break without noticing.

Measured with a refusal-onset detector over StrongREJECT-small (n=60), identical protocol for all

three models, including a positive control:

| Model | Refusal rate |

|---|---|

| zai-org/GLM-4.7-Flash (unmodified base, positive control) | 93.33 % |

| Q4_K-SPLIT | 1.67 % |

| IQ2_M-SPLIT | 3.33 % |

!Refusal

The control matters. A quantization scoring 0 % refusals proves nothing on its own, because a broken

detector also scores 0 %. Here the same detector fires at 93.3 % on the unmodified base model, so

the low scores are evidence rather than an artifact.

Scope, stated honestly: this is a refusal-onset detector built on lexical markers over the first

400 characters of the reply. It measures whether the model starts refusing. It is not the

StrongREJECT harmfulness grader and it is not a safety evaluation. It supports exactly one claim,

that quantization did not restore refusal behaviour, and nothing beyond that.

Reproducibility notes

• Architecture: Glm4MoeLiteForCausalLM converts to GGUF arch deepseek2 with native MLA. The MTP

head is excluded (--no-mtp) and can be exported separately as a speculative draft.

• The Q8_0 and BF16 conversions came out byte-for-byte identical when produced independently

on an A40 and on an A100 in different datacenters, so the source chain is deterministic.

• imatrix.gguf is included so the recipe can be reproduced or extended.

Known gaps

Published deliberately with these stated rather than hidden:

• Behavioural metrics (GSM8K, tool-calling) were measured against a Q8_0 of the same checkpoint

rather than the BF16. In this model that proxy tracked the BF16-referenced KLD to within 1-9 % with

an identical ranking, but it is a proxy.

• Divergence@32 is an internal proxy set (GSM8K + MBPP + fixed prompts), not the published

Divergence-300@32; absolute values are not comparable to anyone else's.

• Long-context behaviour is untested. The model supports 202k context; nothing here was measured

above 8192.

• A Q8-vs-Q8 weight-space comparison against the unmodified base, which would quantify how much

the abliteration changed, has not been run. The refusal delta above establishes it functionally.

Credits

Base model by huihui-ai, built on

zai-org/GLM-4.7-Flash. Quantized by

cognitivers. mradermacher's ladder was used as the comparison baseline and is a fine choice if

you want a conventional set of sizes.