---

license: apache-2.0

base_model: llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic

base_model_relation: quantized

tags:

- gguf

- rocmfpx

- gemma4

- moe

- uncensored

quantized_from: llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic (safetensors)

---

G4-MeroMero-26B-A4B (uncensored heretic) — ROCmFPX hybrid quant

This is a quantization of llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic

(a Heretic-abliterated version of zerofata/G4-MeroMero-26B-A4B,

a Gemma4-architecture 26B MoE: 128 experts, top-8, ~4B activated).

> [!IMPORTANT]

> You need the ROCmFPX fork of llama.cpp (or a llama.cpp

> build with ROCmFPX support). This file uses the experimental q4_0_rocmfp4_fast (type 101) and

> q8_0_rocmfpx (type 103) weight formats, which stock llama.cpp releases do not understand —

> loading it elsewhere will fail with an unknown tensor type error.

> [!NOTE]

> The main quant is text-only; the original model is multimodal (Gemma4 vision tower).

> This repo includes vision projectors exported from the same source weights:

> G4-MeroMero-26B-A4B-mmproj-F16.gguf (F16) and G4-MeroMero-26B-A4B-mmproj-BF16.gguf (BF16),

> for image input support.

Recipe

| Tensor group | Type | Notes |

|---|---|---|

| Routed experts: blk.N.ffn_gate_up_exps (fused gate+up), blk.N.ffn_down_exps | q4_0_rocmfp4_fast (4.25 bpw) | Gemma4 packs gate/up into one fused tensor |

| Everything else quantizable (attention, router, embeddings) | q8_0_rocmfpx (8.25 bpw) | |

| Norms, biases, per-expert scales (*.scale), router scales | f32 (untouched) | |

• Total size: ~14.0 GB (4.64 bpw) vs 50.5 GB BF16 intermediate

How it was made

# 1. HF weights -> BF16 GGUF (from the ROCmFPX fork)
python3 convert_hf_to_gguf.py G4-MeroMero --outfile G4-MeroMero-26B-A4B-BF16.gguf --outtype bf16
# vision projector, exported from the same source weights (add --outtype f16 for the F16 variant):
python3 convert_hf_to_gguf.py G4-MeroMero --mmproj --outfile G4-MeroMero-26B-A4B-mmproj-BF16.gguf

# 2. hybrid quant (CPU-only build works fine for quantization)
llama-quantize \
  --tensor-type "ffn_(gate_up|down)_exps.weight=q4_0_rocmfp4_fast" \
  G4-MeroMero-26B-A4B-BF16.gguf \
  G4-MeroMero-26B-A4B-Q8_0_ROCMFPX-Q4FAST-experts.gguf Q8_0_ROCMFPX

Usage

# build ROCmFPX for your GPU (see the repo README; e.g. Strix Halo):
env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh

./build-strix-rocmfp4/bin/llama-cli \
  -m G4-MeroMero-26B-A4B-Q8_0_ROCMFPX-Q4FAST-experts.gguf \
  --mmproj G4-MeroMero-26B-A4B-mmproj-F16.gguf \
  -dev Vulkan0 -ngl 999 -fa on --jinja

Benchmarks

> [!NOTE]

> All Benchmarks were run on an AI max 395+ with 128GB RAM

| Depth | pp512 t/s | tg128 t/s |

|---|---|---|

| 4096 | 1038.70 ± 6.14 | 42.91 ± 0.24 |

| 16384 | 763.98 ± 8.60 | 40.84 ± 0.10 |

| 65536 | 383.38 ± 5.45 | 34.79 ± 0.27 |

Quality comparison vs BF16 source (perplexity / KLD): TBD.

Attribution & license

• Quantized from: llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic (safetensors, converted to GGUF in-house)
• Base model: zerofata/G4-MeroMero-26B-A4B; abliteration via Heretic
• License: Apache-2.0 (inherited)
• Quant formats by the ROCmFPX project