Koshkasa/Vortex5_G4-Midnight-Macaw-26B-A4B-MXFP4_MOE-GGUF overview

---

license: apache-2.0

base_model:

• Vortex5/G4-Midnight-Macaw-26B-A4B

library_name: ik_llama.cpp

pipeline_tag: text-generation

tags:

• gguf
• quantized
• llama.cpp
• roleplay
• mxfp4

quantized_by: Koshkasa

base_model_relation: quantized

---

What's that?

MXFP4_MOE quantization of Vortex5/G4-Midnight-Macaw-26B-A4B

Use case

Running the model on consumer-grade Blackwell GPUs (due to native fp4 support).

Running on other hardware IS POSSIBLE. I lack conclusive data to say if it's a good idea.

Benchmarks

|Measurement|Q4_K_S (imatrix)|MXFP4_MOE (default, 8 exps)|delta|MXFP4_MOE (16 exps)|delta|MXFP4_MOE (24 exps)|delta|

|---|---|---|---|---|---|---|---|

|pp512|3067.83|3567.72|+16.29%|2783.90|-9.26%|TBD|TBD|

|tg128|108|89.87|-16.79%|76.43|-29.23%|TBD|TBD|

|pp65536|1748.64|2219.20|+26.91%|1789.84|+2.36%|TBD|TBD|

|tg1024|87.81|88.97|+1.20%|67.38|-23.26%|TBD|TBD|

|pp65536 + tg1024 wall time (s)|49.14|41.03|-16.5%|51.81|+5.43%|TBD|TBD|

|hellaswag% (s=42, n=400, greedy)|56.5%|61.00%|+7.96%|64.00%|+13.27%|63.5%|+12.39%|

|hellaswag (s=42, n=400, greedy)|226|244|+18|256|+30|254|+28|

|winogrande% (n=1267, greedy)|59.51%|57.38%|-3.58%|64.17%|+7.83%|66.45%|+11.66%|

|winogrande (n=1267, greedy)|754|727|-27|813|+59|842|+88|

|Size (bytes)|15464827008|14727180640|-4.77%|14727180640|-4.77%|14727180640|-4.77%|

tested on: --temp 0 -ngl 999 -b 512 -ub 512 -ctk q8_0 -ctv q8_0

pp65536/tg1024 test for Q4_K_S was run at -ngl 29 instead - it did not fit otherwise.

pp65536/tg1024 test for MXFP4_MOE(16 exps) was run at -ngl 30 instead - it did not fit otherwise.

Conclusion

MXFP4_MOE allows either Q4_K_S intelligence at -16.5% wall time, or higher than Q4_K_S intelligence at the cost of +5.43% wall time (with experts/tok override to 16), while being of smaller file size.

Is it worth it? I don't know. At this point, my particular config isn't compute-limited, but rather bottlenecked by PCI-E. WYSIWYG.

Disclosure

My only contribution is compute. This is not my merge. Have fun.

Cheers

Google - the base model.

Vortex5 - for the merge effort.

Everyone whose finetunes were included in the merge!