---

license: apache-2.0

language:

• en

pipeline_tag: text-generation

library_name: gguf

base_model:

• KSP-NMAI/Boris-125M-Instruct

quantized_by: KSP-NMAI

datasets:

• tatsu-lab/alpaca

tags:

• boris
• nmai
• gpt2
• gguf
• llama.cpp
• 125M

---

Boris-125M-Instruct-GGUF

GGUF quantizations of KSP-NMAI/Boris-125M-Instruct for

llama.cpp and compatible runtimes

(llama-server, LM Studio, koboldcpp, Jan).

The original safetensors weights live in the

base repo — use those for finetuning or

for any PyTorch-based runtime. GGUF is inference-only.

Which file should I pick?

Use Q8_0, or F16 if you want the exact reference weights.

> ⚠️ Not recommended: IQ1_S. Included for completeness only.

TQ1_0, TQ2_0 and Q1_0 were built and tested but produced unusable output at this scale, so they are deliberately not published here.

Boris-125M is a small model, and quantization behaves differently at this

scale than it does for 7B+ models. The token embedding table is a large fraction

of the parameters and is kept at high precision by llama.cpp, which sets a hard

floor on file size. The practical result: every file here is between

49 MB and 240 MB. Dropping from Q8_0 to IQ1_S saves you a few

tens of megabytes while degrading output substantially. The aggressive quants

are provided for completeness, not because they are a good trade.

Files

| File | Quant | Size | Notes |

|---|---|---|---|

| Boris-125M-Instruct-F16.gguf | F16 | 240M | Reference. Unquantized conversion of the safetensors weights. |

| Boris-125M-Instruct-BF16.gguf | BF16 | 240M | Reference, bfloat16. |

| Boris-125M-Instruct-Q8_0.gguf | Q8_0 | 130M | Effectively lossless. Recommended. |

| Boris-125M-Instruct-Q6_K.gguf | Q6_K | 101M | Near-lossless. |

| Boris-125M-Instruct-Q5_K_M.gguf | Q5_K_M | 94M | Very good quality. |

| Boris-125M-Instruct-Q5_K_S.gguf | Q5_K_S | 91M | |

| Boris-125M-Instruct-Q5_1.gguf | Q5_1 | 96M | |

| Boris-125M-Instruct-Q5_0.gguf | Q5_0 | 91M | |

| Boris-125M-Instruct-Q4_K_M.gguf | Q4_K_M | 87M | Standard 4-bit default for larger models. |

| Boris-125M-Instruct-Q4_K_S.gguf | Q4_K_S | 82M | |

| Boris-125M-Instruct-Q4_1.gguf | Q4_1 | 86M | |

| Boris-125M-Instruct-Q4_0.gguf | Q4_0 | 81M | |

| Boris-125M-Instruct-IQ4_NL.gguf | IQ4_NL | 80M | |

| Boris-125M-Instruct-IQ4_XS.gguf | IQ4_XS | 78M | |

| Boris-125M-Instruct-Q3_K_L.gguf | Q3_K_L | 84M | |

| Boris-125M-Instruct-Q3_K_M.gguf | Q3_K_M | 77M | |

| Boris-125M-Instruct-Q3_K_S.gguf | Q3_K_S | 70M | |

| Boris-125M-Instruct-IQ3_M.gguf | IQ3_M | 74M | |

| Boris-125M-Instruct-IQ3_S.gguf | IQ3_S | 70M | |

| Boris-125M-Instruct-IQ3_XS.gguf | IQ3_XS | 69M | |

| Boris-125M-Instruct-IQ3_XXS.gguf | IQ3_XXS | 61M | |

| Boris-125M-Instruct-Q2_K.gguf | Q2_K | 67M | |

| Boris-125M-Instruct-Q2_K_S.gguf | Q2_K_S | 62M | |

| Boris-125M-Instruct-IQ2_M.gguf | IQ2_M | 59M | |

| Boris-125M-Instruct-IQ2_S.gguf | IQ2_S | 58M | |

| Boris-125M-Instruct-IQ2_XS.gguf | IQ2_XS | 54M | |

| Boris-125M-Instruct-IQ2_XXS.gguf | IQ2_XXS | 52M | |

| Boris-125M-Instruct-IQ1_M.gguf | IQ1_M | 50M | |

| Boris-125M-Instruct-IQ1_S.gguf | IQ1_S | 49M | ⚠️ Broken — incoherent on a greedy smoke test. |

All quantizations below 8-bit were produced with an importance matrix

calibrated on 100 chunks of held-out data drawn from the model's own training

mixture (60% fineweb-edu / 40% dclm).

Usage

# straight from the Hub
llama-server -hf KSP-NMAI/Boris-125M-Instruct-GGUF:Q8_0 --jinja

# or a local file
llama-server -m Boris-125M-Instruct-Q8_0.gguf --jinja

The Alpaca chat template is embedded in every file, so --jinja applies the

correct prompt format automatically.

Prompt format

### Instruction:
{your instruction}

### Response:

Limitations

This is a very small instruction-tuned model. It will produce text that is

frequently inaccurate, inconsistent, or offensive, and has received no

alignment or safety tuning beyond supervised fine-tuning on Alpaca. Do not rely

on it for factual information or deploy it without supervision.

License

Apache 2.0. Copyright 2026 Joseph Jones. See the

base repository for the full notice.