---

license: apache-2.0

language:

• en

pipeline_tag: text-generation

datasets:

• HuggingFaceFW/fineweb-edu
• HuggingFaceTB/smol-smoltalk

tags:

• mixture-of-experts
• moe
• deepseek
• multi-head-latent-attention
• mla
• gguf
• from-scratch
• tiny
• chat
• instruction-tuned

base_model: edededdy/ShallowSeek-mini-it

base_model_relation: quantized

library_name: gguf

quantized_by: edededdy

---

ShallowSeek-mini-it-GGUF

GGUF builds of ShallowSeek-mini-it for llama.cpp, Ollama and LM Studio. The PyTorch weights (model.safetensors), config and tokenizer are in the main repo.

ShallowSeek-mini-it is the instruction-tuned (chat) version of ShallowSeek-mini-base, a tiny DeepSeek-V3-style Mixture-of-Experts model trained from scratch on a laptop CPU. It has 11.0M parameters, of which 6.3M are active per token.

Part of the ShallowSeek family. Not affiliated with DeepSeek.

It answers in a chat format and stops when it's done, but it's still an 11M-parameter model: answers are fluent and on-topic but often wrong or vague, and it invents facts confidently. It's an educational and research artifact; don't use it for anything that matters.

Chat format

<|system|>You are a helpful assistant.<|end|><|user|>What causes the seasons?<|end|><|assistant|>

The model writes its reply and ends it with <|end|>. The GGUF embeds this template, and the GGUF repo includes Ollama template and params files.

Fine-tuning (SFT)

| | |

|---|---|

| Base model | ShallowSeek-mini-base (pretrained on 286M FineWeb-Edu tokens) |

| Conversations | 16,347 (15,530 train / 817 validation), 1.62M assistant tokens trained on |

| Grounded Q&A (~36% of assistant tokens) | ~10.6K single-turn question/answer pairs generated by Qwen3.8-9B from FineWeb-Edu documents in ShallowSeek's own pretraining data, so answers concern text the model saw. About 2% of pairs still refer to "the document"; these were left in |

| General chat (~64%) | 5,684 conversations from HuggingFaceTB/smol-smoltalk: all 1,994 everyday-conversations (small talk, multi-turn), plus openhermes, smol-constraints and smol-magpie-ultra-short (capped at 15% of assistant tokens), filtered to remove code and to fit 1,024 tokens |

| Loss | On assistant replies and their closing <|end|> only; prompts and padding are masked |

| Schedule | 1942 steps, batch 16, 2 epochs, AdamW, LR 1.5e-4 → 1.5e-5 cosine (about 10% of the pretraining peak) |

| MoE routing | Load-balancing biases frozen at their pretrained values during SFT, with padding excluded from the expert-load statistics |

Validation loss on 817 held-out conversations (assistant tokens only): 3.779 → 2.949.

Evaluation: base vs IT

Both are scored on the same raw-text prompts, without the chat template, so the numbers are directly comparable. Chance on MMLU is 25%.

| Metric | Base | IT |

|---|---|---|

| Probes top-5 | 28.6% | 35.7% |

| Mean answer log-prob | -5.68 | -5.40 |

| True-vs-false pairs won | 58.3% | 66.7% |

| MMLU (letter) | 24.9% | 25.6% |

| MMLU (cloze) | 25.5% | 25.6% |

| MMLU biology & medicine (cloze) | 27.8% | 27.7% |

SFT mainly teaches format, turn-taking and when to stop, and MMLU is essentially unchanged. The factual probes improved slightly, plausibly because the grounded Q&A restates facts from documents the model pretrained on. With 14 probes and 12 pairs, treat that as a modest effect rather than a large one.

Usage

Ollama (uses the bundled chat template):

ollama run hf.co/edededdy/ShallowSeek-mini-it-GGUF:Q8_0

llama.cpp:

llama-cli -m ShallowSeek-mini-it-q8_0.gguf -cnv --temp 0.7 --top-p 0.9 --repeat-penalty 1.15

PyTorch: see chat.py in the training code:

python chat.py --ckpt mini-sft.pt --temperature 0.7 --rep-penalty 1.15

Files

| File | Contents |

|---|---|

| model.safetensors | PyTorch weights (including the MTP module), our own parameter names |

| config.json | ModelConfig for deepseek_moe |

| tokenizer.json | Byte-level BPE tokenizer (Hugging Face tokenizers format) |

| chat_template.jinja | Chat template (Jinja), the same one embedded in the GGUF |

| tokenizer_config.json | Special tokens (<|end|> = end of turn), chat template, context length |

| ShallowSeek-mini-it-f16.gguf | GGUF (f16) for llama.cpp / Ollama / LM Studio |

| ShallowSeek-mini-it-q8_0.gguf | GGUF (q8_0) for llama.cpp / Ollama / LM Studio |

Acknowledgements

• Architecture after DeepSeek-V3 (DeepSeek-AI, 2024).
• Pretraining data: FineWeb-Edu (Hugging Face, ODC-BY 1.0).
• Chat data: smol-smoltalk (Hugging Face, Apache-2.0).
• Synthetic Q&A written by Qwen3.8-9B.

Card generated 2026-10-01 by prepare_release.py.