Koshkasa/TheDrummer_Big-Alice-28B-v1-mix_IQ3_K-GGUF overview

---

license: mit

base_model:

• TheDrummer/Big-Alice-28B-v1

library_name: ik_llama.cpp

pipeline_tag: text-generation

tags:

• gguf
• quantized
• ik_llama.cpp
• iq3_k
• 3 bit
• trellis
• roleplay
• mixed precision

quantized_by: Koshkasa

base_model_relation: quantized

---

What's that?

The goal: Make a mixed precision quant of TheDrummer/Big-Alice-28B-v1 using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit in 16GB VRAM with at least 16k tk of KV cache and system overhead.

The result: Mixed precision quantization of TheDrummer/Big-Alice-28B-v1 utilizing higher precision 3-bit quants for ffn and 4-bit trellis for most attention tensors.

quantized with ik_llama.cpp build: 9d07d868

incompatible with mainline llama.cpp

Layout

| Layer | Dims | Dims | Quant |

| --- | --- | --- | --- |

| token\_embd | 5120 | 131072.0 | iq4\_kt |

| | | | |

| | blk| 40| |

| attn\_k | 5120 | 1024 | iq4\_kt |

| attn\_norm | 5120 | 1 | f32 |

| attn\_q | 5120 | 4096 | iq4\_kt |

| attn\_v | 5120 | 1024 | iq4\_kt |

| attn\_output | 4096 | 5120 | iq4\_k |

| ffn\_down | 14336 | 5120 | iq3\_k |

| ffn\_gate | 5120 | 14336 | iq3\_k |

| ffn\_norm | 5120 | 1 | f32 |

| ffn\_up | 5120 | 14336 | iq3\_k |

| | | | |

| output | 5120 | 131072 | iq5\_k |

| output\_norm | 5120 | 1 | f32 |

Conclusion

It works! Kinda. KV cache for upscaled Apriel is still tremendous, and I can only fit 20960tk in q4_0/q4_0 with system overhead. Candidly, I didn't notice too much difference from Snowpiercer, which outputs a very similar quality text without eating a neighbourhood worth of VRAM to produce it.

Regardless, since the quant was made, it is provided. WYSIWYG.

Cheers

SillyTilly - for the base model.

TheDrummer - for the finetuned model.

mradermacher - for the imatrix + the myriad of quants we all benefit from.

ikawrakow and contributors of ik_llama.cpp - I probably misused your ingenious creation.