---

license: apache-2.0

language:

• en
• code

tags:

• moe
• mixtral
• mergekit
• agentic
• tool-calling
• coding
• python
• code-reasoning
• code-editing
• function-calling
• smol

base_model: mistralai/Ministral-3-3B-Instruct-2512

pipeline_tag: text-generation

---

Le Chaton Slim V0.1 — 23.3B Mixtral-MoE Agentic/Coding Model

A Mixtral-style mixture-of-experts model built by fine-tuning 10 experts from

Ministral-3B-Instruct (text-only, vision stripped) and merging with

mergekit-moe. Intended as a coding

assistant and agentic-tool-use backbone.

⚠️ EXPERIMENTAL. Model quality evaluation is pending. This is a hobby

project by someone who barely knows what they're doing. It might be terrible.

If you're brave enough to test it, please drop notes 😅

Quick facts

| | |

|---|---|

| Architecture | Mixtral-style MoE |

| Parameters | 23.3B (much smaller than planned — MoE layers replaced, vision encoder stripped) |

| Base model | mistralai/Ministral-3-3B-Instruct-2512 (text-only) |

| Experts | 10 routed, top-2 active per token |

| Training | Full fine-tune per expert, ~25k samples each (I'm broke) |

| Context length | 128k (Ministral native) |

| Merge tool | mergekit-moe, hidden gate mode |

| Format | Ministral chat template ([INST]...[/INST]) |

Why it's only 23.3B instead of ~40B

Honestly I aimed for 35-40B. The MoE merge strips the base model's dense MLP

layers and replaces them with a router + shared expert MLPs. On top of that I

ripped out the vision encoder since this model doesn't need images. Result:

smaller. If I make a V2 I might bump the expert count and switch to DeepSeek

MoE (shared + routed experts) to hit the size target.

Experts

Each expert was full fine-tuned on its own dataset mix so the router has

something meaningful to route toward. Only 25k samples/epoch because GPU time

costs money and I'm not made of it.

| # | Expert | Specialty |

|---|---|---|

| 1 | Captain | General instruction following + safety boundaries |

| 2 | Python Coding | Core Python SFT (Magicoder OSS/Evol-Instruct) |

| 3 | Verified Python | Execution-filtered code + OPC educational instruct |

| 4 | Code Reasoning | Step-by-step algorithm design, CoT (OpenCodeReasoning + OpenThoughts) |

| 5 | Tool Calling | Function calling + structured JSON (Hermes-fc + ToolACE + xLAM) |

| 6 | Agentic Workflows | Multi-step agent planning, ReAct loops (Hermes-agent + Nemotron-agentic) |

| 7 | Code Editing | Diff generation, refactoring (commitpackft) |

| 8 | Code Review | Code critique + iterative improvement (CodeFeedback) |

| 9 | General Reasoning | Math, logic, non-code problem solving |

| 10 | Safety Net | Untuned base model — catch-all for chitchat, greetings, refusals |

The captain (expert #1) donates self-attention and layer norms to the whole

model. The safety net (expert #10) is the raw bf16 base with no fine-tuning at

all — it catches prompts that don't match any specialty so you don't get the

tool-calling expert trying to process "hello".

Limitations (there are many)

• Not evaluated. I ran a local smoke test (3k samples/expert, q8_0 GGUF)

and it spits out text, but I haven't done any real benchmarks. The production

run trains at 25k or 75k samples per expert and I haven't tested the final

merge yet.

• Low sample counts. 25k samples per expert is tiny by modern standards.

Quality is likely mediocre.

• Training quality unknown. The local smoke run used 3k samples each

(seq_len 1024) — results were... not great. May be the tiny dataset, may be

the idea itself is bad. Production run on a RunPod A40 might fix it, might

not. ¯\\_(ツ)_/¯

• No RLHF/DPO. This is SFT-only. No preference optimization, no reward

modeling. It will not be aligned beyond what the datasets provide.

• No vision. The vision encoder was completely stripped. This is text-only.

Planned V2 (if this isn't terrible)

• DeepSeek MoE architecture: shared experts (always active) + routed

experts, which should handle overlapping domains better.

• ~14 experts instead of 10, for more granular specialization.
• Proper eval harness before shipping.
• Maybe a separate "fanfiction" ( ͡° ͜ʖ ͡°) creative writing fine-tune if I feel like

it. No promises.

How to use

This is a standard transformers model with a Mixtral architecture. The chat

template is the Ministral format:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "HavocK1/Le-Chaton-Slim-MoE",
    torch_dtype="auto",
    device_map="auto",
)

tokenizer = AutoTokenizer.from_pretrained("HavocK1/Le-Chaton-Slim-MoE")

messages = [
    {"role": "user", "content": "Write a Python function to reverse a linked list."},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For GGUF (llama.cpp):

llama-server le_chaton_slim.gguf --n-gpu-layers 99

Datasets

The training datasets (deduped JSONL) are available at

HavocK1/Le-Chaton-Slim-Experts.

Source datasets used:

• HuggingFaceTB/smoltalk, HuggingFaceH4/no_robots, Magpie-Align/Magpie-Pro-300K-Filtered, teknium/OpenHermes-2.5, HuggingFaceH4/ultrachat_200k
• ise-uiuc/Magicoder-OSS-Instruct-75K, ise-uiuc/Magicoder-Evol-Instruct-110K
• bigcode/self-oss-instruct-sc2-exec-filter-50k, OpenCoder-LLM/opc-sft-stage2
• nvidia/OpenCodeReasoning, open-thoughts/OpenThoughts-114k
• NousResearch/hermes-function-calling-v1, Team-ACE/ToolACE, Salesforce/xlam-function-calling-60k
• lambda/hermes-agent-reasoning-traces, nvidia/Nemotron-SFT-Agentic-v2
• bigcode/commitpackft
• m-a-p/CodeFeedback-Filtered-Instruction, m-a-p/Code-Feedback
• QuixiAI/dolphin-r1, HuggingFaceTB/smoltalk (NuminaMath-CoT)

Credits

Built with:

• Ministral-3B

(Mistral AI)

• mergekit (Arcee AI)
• HuggingFace transformers +

TRL + PEFT

• All the amazing dataset authors above 🙏
• An A40 GPU on RunPod that I paid for with actual money
• Shout out to my homies Mistral 3.5 128B, GLM 5.2 and DS V4 pro. Couldn't have done it without them