Koshkasa/ShyliaSafetensors_Ariel-Alloy-V1-24B-Heretic-IQ4_K_M.GGUF overview

---

license: apache-2.0

base_model:

• ShyliaSafetensors/Ariel-Alloy-V1-24B-Heretic

library_name: ik_llama.cpp

pipeline_tag: text-generation

tags:

• gguf
• quantized
• ik_llama.cpp
• roleplay
• trellis
• mixed precision

quantized_by: Koshkasa

base_model_relation: quantized

---

What's that?

The goal: Make a quality quant of ShyliaSafetensors/Ariel-Alloy-V1-24B-Heretic using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit into 16gb VRAM with KVO, accounting for system overhead.

This time the wheel was reinvented one spoke at a time - blocks in proximity to input and output are treated with premium precision, while the more tolerant middle blocks are compressed in a more aggressive 4-bit precision. Attention is kept high in all blocks.

This is in line of how popular Q4_K_S/M quants are made for Mistral 24B, except we use IQ_K quants and utilize trellis for middle ffn_up and ffn_gate blocks.

Three versions were cooked. Two lost marginally to Q4_K_S in hellaswag and winogrande, though perhaps they would perform better in NIHS-tests (due to higher attn), which I don't have the patience to set up. For what they offered, they were slow - they had even more trellis, and it shows.

Option three, which is this one, compromised some attn precision for better protection of ffn_down layers. Lacking a conventional naming scheme for these, I went with IQ4_K_M. It looks stupid. I don't know how to fix that.

The result: Mixed precision quantization of ShyliaSafetensors/Ariel-Alloy-V1-24B-Heretic

quantized with ik_llama.cpp build: 9d07d868

incompatible with mainline llama.cpp

<details> <summary><strong>Layout: IQ4_K_M</h4></summary>

| Layer | Dims | Dims | Quant |

|--|--|--|--|

| token\_embd | 5120 | 131072.0 | iq5\_ks |

<div style="display: flex; flex-wrap: wrap; gap: 20px; justify-content: center; ">

<div style="flex: 1; min-width: 200px;">

<h4>Blocks 0, 1, 38, 39</h4>

<table>

<thead>

<tr><th>Layer</th><th>Dims</th><th>Dims</th><th>Quant</th></tr>

</thead>

<tbody>

<tr><td>attn_k</td><td>5120</td><td>1024</td><td><strong>iq6_k</strong></td></tr>

<tr><td>attn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>

<tr><td>attn_q</td><td>5120</td><td>4096</td><td><strong>iq6_k</strong></td></tr>

<tr><td>attn_v</td><td>5120</td><td>1024</td><td><strong>iq6_k</strong></td></tr>

<tr><td>ffn_down</td><td>32768</td><td>5120</td><td><strong>iq6_k</strong></td></tr>

<tr><td>ffn_gate</td><td>5120</td><td>32768</td><td><strong>iq6_k</strong></td></tr>

<tr><td>ffn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>

<tr><td>ffn_up</td><td>5120</td><td>32768</td><td><strong>iq6_k</strong></td></tr>

<tr><td>attn_output</td><td>4096</td><td>5120</td><td><strong>iq6_k</strong></td></tr>

</tbody>

</table>

</div>

<div style="flex: 1; min-width: 200px;">

<h4>Blocks 2, 3, 37, 34-37</h4>

<table>

<thead>

<tr><th>Layer</th><th>Dims</th><th>Dims</th><th>Quant</th></tr>

</thead>

<tbody>

<tr><td>attn_k</td><td>5120</td><td>1024</td><td><strong>iq6_k</strong></td></tr>

<tr><td>attn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>

<tr><td>attn_q</td><td>5120</td><td>4096</td><td><strong>iq5_k</strong></td></tr>

<tr><td>attn_v</td><td>5120</td><td>1024</td><td><strong>iq6_k</strong></td></tr>

<tr><td>ffn_down</td><td>32768</td><td>5120</td><td><strong>iq5_k</strong></td></tr>

<tr><td>ffn_gate</td><td>5120</td><td>32768</td><td><strong>iq5_ks</strong></td></tr>

<tr><td>ffn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>

<tr><td>ffn_up</td><td>5120</td><td>32768</td><td><strong>iq5_ks</strong></td></tr>

<tr><td>attn_output</td><td>4096</td><td>5120</td><td><strong>iq5_k</strong></td></tr>

</tbody>

</table>

</div>

<div style="flex: 1; min-width: 200px;">

<h4>Blocks 6–33</h4>

<table>

<thead>

<tr><th>Layer</th><th>Dims</th><th>Dims</th><th>Quant</th></tr>

</thead>

<tbody>

<tr><td>attn_k</td><td>5120</td><td>1024</td><td><strong>iq5_k</strong></td></tr>

<tr><td>attn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>

<tr><td>attn_q</td><td>5120</td><td>4096</td><td><strong>iq5_ks</strong></td></tr>

<tr><td>attn_v</td><td>5120</td><td>1024</td><td><strong>iq6_k</strong></td></tr>

<tr><td>ffn_down</td><td>32768</td><td>5120</td><td><strong>iq4_k</strong></td></tr>

<tr><td>ffn_gate</td><td>5120</td><td>32768</td><td><strong>iq4_kt</strong></td></tr>

<tr><td>ffn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>

<tr><td>ffn_up</td><td>5120</td><td>32768</td><td><strong>iq4_kt</strong></td></tr>

<tr><td>attn_output</td><td>4096</td><td>5120</td><td><strong>iq5_k</strong></td></tr>

</tbody>

</table>

</div>

| Layer | Dims | Dims | Quant |

|--|--|--|--|

| output | 5120 | 131072 | iq6\_k |

| output\_norm | 5120 | 1 | f32 |

</div>

</details>

using imatrix by mradermacher

Rationale

Hopefully the beefed up attention will help over contexts this quant is intended to run (16k-32k). It is not statistically dumber than its main competitor, Q4_K_S, outsmarted by 3-4 responses over 1267 winogrande and 800 hellaswag questions (s = 123). While the difference is not statistically significant, it's there. A NIHS test would probably be this quant's stronger suit, but I lack quality data to test it. Experimental quant. WYSIWYG.

Cheers

MistralAI - the beloved base model(s).

ikawrakow and contributors of ik_llama.cpp - I probably misused your wonderful creation.

ShyliaSafetensors - for the merge effort.

Everyone whose finetunes were included in the merge!

mradermacher - for the imatrix + the myriad of quants we all benefit from.