Koshkasa/TheDrummer_Big-Alice-28B-v1-IQ4_KT-GGUF overview
What's that? The goal : Make a competitive 4 bit quality quant of TheDrummer/Big Alice 28B v1 using SOTA quant types from ik llama.cpp, allowing the resulting …
Runs locally from ~14.50 GB disk (16 GB VRAM class GPUs with llama.cpp / guIDE).
Repository Files & Downloads
| File | Type | Quantization | Size | Link |
|---|---|---|---|---|
| TheDrummer_Big-Alice-28B-v1-IQ4_KT.gguf | GGUF | IQ4_KT | 14.50 GB | Download |
Model Details
| Model ID | Koshkasa/TheDrummer_Big-Alice-28B-v1-IQ4_KT-GGUF |
|---|---|
| Author | Koshkasa |
| Pipeline | text-generation |
| License | mit |
| Base model | TheDrummer/Big-Alice-28B-v1 |
| Last modified | 2026-07-28T10:05:57.000Z |
Model README
---
license: mit
base_model:
- TheDrummer/Big-Alice-28B-v1
library_name: ik_llama.cpp
pipeline_tag: text-generation
tags:
- gguf
- quantized
- ik_llama.cpp
- iq4_kt
- 4 bit
- trellis
- roleplay
- mixed precision
quantized_by: Koshkasa
base_model_relation: quantized
---
What's that?
The goal: Make a competitive 4-bit quality quant of TheDrummer/Big-Alice-28B-v1 using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit into 16gb VRAM without KVO, accounting for system overhead.
The result: Mixed precision quantization of TheDrummer/Big-Alice-28B-v1 utilizing 4-bit trellis quants for FFN tensors that is theoretically smarter and faster than IQ4_XS.
quantized with ik_llama.cpp build: 9d07d868
incompatible with mainline llama.cpp
Layout
| Layer | Dims | Dims | Quant |
| --- | --- | --- | --- |
| token\_embd | 5120 | 131072.0 | iq4\_k |
| | | | |
| | blk| 40| |
| attn\_k | 5120 | 1024 | iq5\_ks |
| attn\_norm | 5120 | 1 | f32 |
| attn\_q | 5120 | 4096 | iq5\_ks |
| attn\_v | 5120 | 1024 | iq6\_k |
| attn\_output | 4096 | 5120 | iq5\_k |
| ffn\_down | 14336 | 5120 | iq4\_kt |
| ffn\_gate | 5120 | 14336 | iq4\_kt |
| ffn\_norm | 5120 | 1 | f32 |
| ffn\_up | 5120 | 14336 | iq4\_kt |
| | | | |
| output | 5120 | 131072 | iq6\_k |
| output\_norm | 5120 | 1 | f32 |
Conclusion
In retrospect, it probably wasn't the best idea for 16GB VRAM - the PCI-E travel overhead from KV cache in RAM is MASSIVE for upscaled Apriel (especially for my 8-lane 5060 Ti.). This one remains an ok option at 20gb+ VRAM, or on 16gb GPUs with a less impotent PCI-E config. A smaller 3.44 bpw ffn + trellis 4.0 bpw attention quant is in the overn for strictly 16gb inference.
Regardless, this one is provided as is. WYSIWYG.
Cheers
SillyTilly - for the base model.
TheDrummer - for the finetuned model.
mradermacher - for the imatrix + the myriad of quants we all benefit from.
ikawrakow and contributors of ik_llama.cpp - I probably misused your ingenious creation.
Run Koshkasa/TheDrummer_Big-Alice-28B-v1-IQ4_KT-GGUF with guIDE
Download guIDE — the AI-native code editor with local LLM inference and 69 built-in tools.
Source: Hugging Face · Compare models