---

license: apache-2.0

base_model:

• Naphula/Slimaki-Tavern-24B-v1.3

library_name: ik_llama.cpp

pipeline_tag: text-generation

tags:

• gguf
• quantized
• ik_llama.cpp
• 4-bit
• trellis
• roleplay

quantized_by: Koshkasa

base_model_relation: quantized

---

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

This is a IQ4_KT quantization of Naphula/Slimaki-Tavern-24B-v1.3, created specifically to challenge mainline IQ4_XS quants by being smaller and not being dumber.

Quant Details

iq4_kt: attn_q, attn_v, attn_k, ffn_down, ffn_gate, ffn_up, attn_output (280)

iq4_k: token_embd (1)

iq6_k: output (1)

imatrix by mradermacher

quantized with ik_llama.cpp build: 9d07d868

incompatible with mainline llama.cpp

only tested with ik_llama.cpp (though I expect croco.cpp to work as well?)

Comparison:

IQ4_XS by mradermacher:

winogrande: 74.3489 ± 1.2274 (942/1267)

hellaswag (n=400, s=42): 82.25%

with K q5_0, V q4_1:

pp512: 1457.63 ± 157.65

pp32768: 1108.26 ± 5.68

tg128: 31.27 ± 0.79

tg512: 31.54 ± 0.16

size: 12758946304 bytes

this IQ4_KT:

winogrande: 74.7435 ± 1.2211 (947/1267) (+0.53%, statistically insignificant)

hellaswag (n=400, s=42): 83.25% (+1.2%, statistically insignificant)

with K q5_0, V q4_1:

pp512: 1168.07 ± 30.28 (19.8% slower)

pp32768: 955.13 ± 4.38 (13.8% slower)

tg128: 32.39 ± 0.22 (3.58% faster)

tg512: 31.52 ± 0.26 (0.07% slower)

size: 12070818560 bytes (5.39% smaller) (-656 MB)

In conclusion

-Is it smarter?

Not really.

-Is it slower?

Yeah.

-Is it smaller?

Yeah. 656 MB is about 12k extra context at K(Q5_0) V(Q4_1).

-Is it worth it?

No idea. I made it for personal use to squeeze some extra context in without losing brainpower. I can take the PP speed hit. WYSIWYG.

-Possible 12G VRAM full fit with KVO?

Probably. A mixed IQ3_KT ffn / IQ4_KT attn would likely fit with some context headroom. It's going to be slower. It's probably not going to be THAT much dumber.

-Why Q5_0/Q4_1?

Reasons

Cheers

MistralAI - the beloved base model(s).

ikawrakow and contributors of ik_llama.cpp - I probably misused your wonderful creation.

Naphula - you cooked a sick one.

Everyone whose finetunes were included in the merge!

mradermacher - for the imatrix + the myriad of quants we all benefit from.