---

language:

• ro
• en

license: cc-by-nc-4.0

base_model: rostlabs/rost-1b-instruct

library_name: gguf

pipeline_tag: text-generation

tags:

• gguf
• llama.cpp
• romanian
• bilingual
• nanochat

---

rost-1b-instruct, GGUF

> ### These files will not load in Ollama, LM Studio, or a stock llama.cpp build

>

> rost is not an architecture llama.cpp knows. llama.cpp compiles architectures in rather

> than loading them dynamically, so support has to be present in the binary. It is not

> upstream yet, and Ollama and LM Studio bundle their own llama.cpp. A stock build will

> refuse these files with unknown model architecture: 'rost'.

>

> A patched build is required. Instructions are below.

Quantisations of rostlabs/rost-1b-instruct,

a 1.4B bilingual Romanian/English model.

Files

| file | size | bits/weight | notes |

| --- | ---: | ---: | --- |

| rost-1b-instruct-Q4_K_M.gguf | 791 MiB | 4.79 | the one to start with |

| rost-1b-instruct-Q6_K.gguf | 1.08 GiB | 6.56 | |

| rost-1b-instruct-Q8_0.gguf | 1.40 GiB | 8.50 | |

| rost-1b-instruct-BF16.gguf | 2.64 GiB | 16.00 | the training precision; no quantisation loss |

Measured on this checkpoint, generating with twelve CPU threads and on one RTX 5070:

| | RTX 5070 | 12 CPU threads |

| --- | ---: | ---: |

| Q4_K_M | 489.7 tok/s | 60.6 tok/s |

| F32 (for scale) | 170.1 tok/s | 14.3 tok/s |

Building a llama.cpp that can read these

There is a fork with the architecture already in it. Three lines:

git clone -b master https://github.com/rostlabs/llama.cpp && cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

Its master is upstream llama.cpp master plus a single commit adding the architecture. Metal

is included automatically on macOS; add -DGGML_CUDA=ON for NVIDIA.

The architecture is one commit; git log -1 in that clone shows what it adds and why.

Then:

./build/bin/llama-cli -m rost-1b-instruct-Q4_K_M.gguf \
  --repeat-penalty 1.1 -p "Care este capitala României?"

Sampling

Use a repetition penalty of 1.1. It is not optional decoration on this model. Measured

over 198 generations of 400 tokens on prompts chosen to provoke the failure:

| decoding | replies that looped | worst repeated 6-gram |

| --- | ---: | ---: |

| greedy | 50% | 44x |

| temperature 0.2, no penalty | 44% | 44x |

| temperature 0.6, no penalty | 22% | 10x |

| temperature 0.6, penalty 1.1 | 0% | 2x |

Stronger penalties suppress repetition further and cost accuracy: at 1.3 the model stops

looping and starts inventing etymologies. 1.1 is the mildest setting that works.

The transformers repository ships these defaults in generation_config.json; llama-cli

needs --repeat-penalty 1.1 passed explicitly.

How faithful are these files?

The port is checked against a second implementation of the forward pass, written from the

safetensors, which agrees with the published model to 1.7e-05:

| | |

| --- | --- |

| prefill, stage by stage | 5.8e-06 to 1.8e-04 |

| single-token decode, top-8 probabilities | 4.9e-05 |

| after a cache rollback | 8.0e-05 |

| two concurrent sequences | 1.8e-05 to 1.5e-04 |

| tokenisation | exact on 8 of 8 strings, including whitespace runs, digit groups and emoji |

| CUDA | top-8 order matches, 3.8e-04 prefill and 2.7e-04 decode |

| Metal | top-8 order matches, 4.3e-05 prefill and 1.2e-04 decode |

On an M3 the Q4_K_M file runs at 1435 t/s prompt and 77 t/s generation.

At BF16 the greedy output matches the reference exactly. At Q4_K_M a token can differ, which

is ordinary quantisation.

A note on file revisions

These files were regenerated on 2026-08-20. Two things the checkpoint does unusually — a

transposed rotary convention, and a QK norm with no learnable weight — turned out to fold into

the weights, so the architecture now uses llama.cpp's ordinary rope and its existing

q_norm/k_norm path instead of carrying special cases. The maths is exact, and the outputs

are unchanged.

The file format changed as a result: **an earlier download will not load in a current build,

and these files will not load in a build from before that change.** If you took a copy on the

day of publication, take it again.

Limitations

Everything in the main model card

applies: it confabulates confidently, its reasoning is near chance, the context is 4,096

tokens, and no safety tuning of any kind has been applied. Read that page before using this.