---

license: apache-2.0

language:

• en

pipeline_tag: text-generation

library_name: gguf

base_model:

• KSP-NMAI/Boris-1.3-75M-Instruct

quantized_by: KSP-NMAI

datasets:

• HuggingFaceTB/smol-smoltalk
• OpenAssistant/oasst1

tags:

• boris
• nmai
• gpt2
• gguf
• llama.cpp
• 75M

---

Boris-1.3-75M-Instruct-GGUF

GGUF quantizations of KSP-NMAI/Boris-1.3-75M-Instruct for

llama.cpp and compatible runtimes

(llama-server, LM Studio, koboldcpp, Jan).

The original safetensors weights live in the

base repo — use those for finetuning or

for any PyTorch-based runtime. GGUF is inference-only.

Which file should I pick?

Use Q8_0, or F16 if you want the exact reference weights.

Boris-1.3-75M is a small model, and quantization behaves differently at this

scale than it does for 7B+ models. Two things set a hard floor on file size:

• The token embedding is 576 × 50304 ≈ 29M weights — 38% of the model — and

llama.cpp keeps it at Q8_0 (~31 MB) in every quantized file here. The output

weights are tied to it, so that 31 MB is unavoidable.

• d_model is 576 and the QKV/FFN-up rows are 1728; neither is divisible by 256,

which k-quant and i-quant blocks require. Those tensors fall back to Q4_0, so

only ffn_down (2304 columns) is ever quantized at the nominal bit width.

The practical result: every file here is between 54 MB and 151 MB. Dropping from

Q8_0 to IQ1_S saves you a few tens of megabytes while degrading output

substantially, and the ternary files are broken outright. The aggressive quants

are provided for completeness, not because they are a good trade.

Files

Perplexity is measured on a fixed 30 × 1024-token slice of the calibration

corpus. It is useful for comparing these rows against each other and nothing else.

| File | Quant | Size | PPL | Notes |

|---|---|---|---|---|

| Boris-1.3-75M-Instruct-F16.gguf | F16 | 151M | 71.04 | Reference. Unquantized conversion of the safetensors weights. |

| Boris-1.3-75M-Instruct-BF16.gguf | BF16 | 151M | 71.04 | Reference, bfloat16. |

| Boris-1.3-75M-Instruct-Q8_0.gguf | Q8_0 | 82M | 71.13 | Effectively lossless. Recommended. |

| Boris-1.3-75M-Instruct-Q6_K.gguf | Q6_K | 78M | 71.05 | Near-lossless. |

| Boris-1.3-75M-Instruct-Q5_K_M.gguf | Q5_K_M | 70M | 71.22 | Very good quality. |

| Boris-1.3-75M-Instruct-Q5_K_S.gguf | Q5_K_S | 67M | 71.24 | |

| Boris-1.3-75M-Instruct-Q5_1.gguf | Q5_1 | 68M | 71.85 | |

| Boris-1.3-75M-Instruct-Q5_0.gguf | Q5_0 | 65M | 71.93 | |

| Boris-1.3-75M-Instruct-Q4_K_M.gguf | Q4_K_M | 67M | 71.71 | Standard 4-bit default for larger models. |

| Boris-1.3-75M-Instruct-Q4_K_S.gguf | Q4_K_S | 64M | 72.08 | |

| Boris-1.3-75M-Instruct-Q4_1.gguf | Q4_1 | 62M | 73.46 | |

| Boris-1.3-75M-Instruct-Q4_0.gguf | Q4_0 | 59M | 74.05 | |

| Boris-1.3-75M-Instruct-IQ4_NL.gguf | IQ4_NL | 59M | 73.56 | |

| Boris-1.3-75M-Instruct-IQ4_XS.gguf | IQ4_XS | 59M | 73.60 | |

| Boris-1.3-75M-Instruct-Q3_K_L.gguf | Q3_K_L | 64M | 71.82 | |

| Boris-1.3-75M-Instruct-Q3_K_M.gguf | Q3_K_M | 61M | 72.84 | |

| Boris-1.3-75M-Instruct-Q3_K_S.gguf | Q3_K_S | 57M | 74.75 | |

| Boris-1.3-75M-Instruct-IQ3_M.gguf | IQ3_M | 59M | 73.29 | |

| Boris-1.3-75M-Instruct-IQ3_S.gguf | IQ3_S | 57M | 73.89 | |

| Boris-1.3-75M-Instruct-IQ3_XS.gguf | IQ3_XS | 57M | 73.89 | |

| Boris-1.3-75M-Instruct-IQ3_XXS.gguf | IQ3_XXS | 57M | 75.57 | |

| Boris-1.3-75M-Instruct-Q2_K.gguf | Q2_K | 57M | 74.75 | |

| Boris-1.3-75M-Instruct-Q2_K_S.gguf | Q2_K_S | 56M | 78.03 | |

| Boris-1.3-75M-Instruct-IQ2_M.gguf | IQ2_M | 56M | 76.42 | |

| Boris-1.3-75M-Instruct-IQ2_S.gguf | IQ2_S | 55M | 78.52 | |

| Boris-1.3-75M-Instruct-IQ2_XS.gguf | IQ2_XS | 55M | 79.67 | |

| Boris-1.3-75M-Instruct-IQ2_XXS.gguf | IQ2_XXS | 55M | 83.35 | |

| Boris-1.3-75M-Instruct-IQ1_M.gguf | IQ1_M | 54M | 91.06 | Still clean English, but loops and confabulates. |

| Boris-1.3-75M-Instruct-IQ1_S.gguf | IQ1_S | 54M | 97.65 | |

| Boris-1.3-75M-Instruct-TQ2_0.gguf | TQ2_0 | 55M | 728.94 | Ternary; grammatical but semantically broken. Curiosity only. |

| Boris-1.3-75M-Instruct-TQ1_0.gguf | TQ1_0 | 54M | 728.94 | Ternary; as above, and gives the same greedy output as TQ2_0. |

All K-quant, IQ and TQ files were produced with an importance matrix calibrated

on 300 chunks of held-out data drawn from the model's own pretraining mixture

(60% fineweb-edu / 40% dclm) plus rendered instruction turns; the matrix itself

is published here as boris-1.3-75m-instruct.imatrix. Q8_0 and the legacy

Q4_0/Q4_1/Q5_0/Q5_1 are plain round-to-nearest. SHA256SUMS covers every file.

Converted and quantized with llama.cpp build b9013 (e48034dfc).

Usage

# straight from the Hub
llama-server -hf KSP-NMAI/Boris-1.3-75M-Instruct-GGUF:Q8_0 --jinja

# or a local file
llama-server -m Boris-1.3-75M-Instruct-Q8_0.gguf --jinja

The Alpaca chat template is embedded in every file, so --jinja applies the

correct prompt format automatically. A system message is folded into the front

of the following instruction, matching how the model was fine-tuned.

Prompt format

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{your instruction}

### Response:

The response begins with a leading space and ends with <|endoftext|> (token 0),

which is also the stop token.

Verification

Every file in this repo was checked to load as arch=gpt2 with the chat template

present, and to complete Alpaca prompts with output that decodes as strict UTF-8

containing zero U+FFFD replacement characters — no mojibake at any quant level,

including the ternary files. The GGUF vocab was diffed against the source

tokenizer.json across accented Latin, Cyrillic, CJK, Hangul, Arabic, Hebrew,

emoji (including ZWJ sequences), zero-width and whitespace-run samples: identical

token ids and byte-exact detokenization.

Limitations

This is a very small instruction-tuned model. It will produce text that is

frequently inaccurate, inconsistent, or offensive, and has received no alignment

or safety tuning beyond supervised fine-tuning. Do not rely on it for factual

information or deploy it without supervision.

License

Apache 2.0. Copyright 2026 Joseph Jones. See the

base repository for the full notice.