---

license: apache-2.0

base_model: Qwen/Qwen3-8B

language:

• en
• de

tags:

• lora
• peft
• function-calling
• task-planning
• moe-orchestration
• gguf
• lumi
• amd-mi250x

pipeline_tag: text-generation

---

qwen3-planner-sft

A LoRA fine-tune of Qwen/Qwen3-8B, trained to act as the

Planner component of MoE Sovereign, a local,

sovereign Mixture-of-Experts LLM orchestrator. Given a user request, the model decomposes it into

1–4 subtasks and emits a single JSON array assigning each subtask to a specialist expert category

or a deterministic MCP tool — nothing else. It is not a general-purpose chat/instruct model;

it has one job, and the training and evaluation below are scoped to that job.

This card documents the training run in detail, including two significant failure modes from an

earlier attempt, because the fixes are only meaningful in that context and because HPC-cluster SFT

failures of this shape (silent, no error, normal-looking loss curve) are easy to reproduce

elsewhere if the causes aren't spelled out.

Model summary

| | |

|---|---|

| Base model | Qwen/Qwen3-8B (dense, 8.19B params, Apache 2.0) |

| Fine-tuning method | LoRA, r=16, α=32, dropout=0.05 |

| LoRA targets | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |

| Trainable params | 43.65M / 8.23B (0.53%) |

| Precision | BF16 (full precision base + LoRA; no quantization during training) |

| Context length trained | 4,096 tokens |

| Formats published | safetensors (merged, BF16, ~16 GB), GGUF F16 (~16 GB), GGUF Q4_K_M (~4.7 GB, this repo) |

| License | Apache 2.0 (inherited from base model; no additional restrictions) |

Intended use

Structured task decomposition for a multi-expert LLM orchestrator. Input: a system prompt fixing

a 16-category expert taxonomy plus a tool catalog (see "Prompt format" below), followed by a user

query. Output: only a JSON array, e.g.:

[{"task": "Explain the difference between TCP and UDP", "category": "technical_support"},
 {"task": "Convert 100 km/h to m/s", "category": "precision_tools",
  "mcp_tool": "unit_convert", "mcp_args": {"value": 100, "from_unit": "km/h", "to_unit": "m/s"}}]

The model is trained against one specific, versioned system prompt

(PLANNER_SYSTEM_PROMPT in the MoE Sovereign codebase, ~2,500 tokens, 16 expert categories + a

22-tool precision catalog). Using it with a different taxonomy or system prompt is expected to

degrade quality — see "Known limitations."

Training data

• 259,829 synthetic (query, plan) pairs, 258,529 train / 1,300 held-out eval

(val_split=0.005).

• 5 system-prompt phrasing variants plus ~15% adversarial/negative samples (queries designed to

probe category-boundary confusion, e.g. arithmetic phrased as a "technical" question that should

still route to precision_tools, not technical_support).

• Teacher model: meta-llama/Meta-Llama-3.1-405B-Instruct (project default for this dataset

generator; Qwen/Qwen3-235B-A22B-Instruct is the documented fallback teacher — the exact teacher

used for this specific merged dataset build was not independently re-verified against the

original generation logs at model-card-writing time, so treat this as "default per the

generation script," not as an independently re-confirmed fact).

• Dataset is internal to the MoE Sovereign project and not currently published.

Training procedure

• Framework: TRL SFTTrainer + PEFT LoRA + DeepSpeed ZeRO-2 (not ZeRO-3 — the 8B model fits

in a single GPU's memory without parameter sharding, so ZeRO-2 is faster and, unlike ZeRO-3,

composes cleanly with LoRA).

• Loss masking: assistant_only_loss=True — loss is computed only over the assistant's JSON

response, not over the system+user portion of the sequence. This is a native TRL 1.4 feature and

is the fix for the truncation failure described below, not an incidental setting.

• Schedule: 3 epochs, micro-batch 4 × grad-accum 4 × 8 GPUs = effective batch 128, cosine LR

schedule, peak LR 2e-4, gradient checkpointing enabled.

• Attention: eager (attn_implementation="eager") — Flash Attention 2 is not available for the

ROCm 7.0 / PyTorch 2.10 build used on this cluster, so attention materializes the full

O(seq²) matrix rather than a fused/tiled kernel. This is the dominant cost factor behind the

per-step latency below, not model size or LoRA overhead.

Hardware: LUMI-G (EuroHPC)

Trained on LUMI-G, the GPU partition of the EuroHPC LUMI supercomputer (CSC, Finland),

under EuroHPC project allocation project_465003058. One small-g node: 8× AMD Instinct

MI250X GCDs (Graphics Compute Dies — each physical MI250X card exposes two GCDs as separate

devices), 512 GB HBM2e, ROCm 7.0, PyTorch 2.10+rocm7.0, executed inside a Singularity container.

Training run and two prior failure modes

This model is the result of the second full training attempt against this dataset. Both

failure modes below are documented here (rather than only in internal project notes) because

they cost real HPC allocation and neither produced an error — both looked like a normal,

converging training run right up until post-hoc inspection.

Attempt 1 — silent truncation of the training target (root-caused, not repeated here):

The first run used max_seq_len=1536, carried over from an earlier, much shorter prompt format

and never re-checked after the system prompt grew to ~2,500 tokens. With

truncation_side="right" and no completion-only loss masking, the assistant's JSON target — which

starts after the ~2,500-token system prompt — fell outside the 1,536-token window for

essentially all 259,829 examples. The loss converged normally (the model was trivially learning to

predict text it was already given in full, i.e. the system prompt prefix), and the run reported no

errors. The resulting model passed a superficial capability check (valid JSON, plausible

categories) purely because the base model already had those capabilities zero-shot — the

fine-tune itself had no measurable effect. Root-caused by tokenizing real training examples

post-hoc and finding the assistant span started at token ≈2,536, far past the 1,536-token cutoff.

Attempt 2, first sub-issue — GPU device-placement bug (fixed): early re-runs (with

max_seq_len corrected) hit repeated out-of-memory errors that looked like a batch-size problem

and were initially treated as one (leading to a QLoRA 4-bit detour that "fixed" the symptom without

being the actual cause). The real cause: the training script never called

torch.cuda.set_device(local_rank) before model loading. Because SFTConfig/TrainingArguments

(which normally performs this via Accelerate) is only constructed after from_pretrained() runs,

all 8 DeepSpeed ranks defaulted to the same device and loaded 8 copies of the model onto GPU 0

while the other 7 GPUs sat idle — visible in an OOM trace showing ~59 GB allocated on one GPU by a

process that should have used ~5 GB of its own. Fixed with a single explicit

torch.cuda.set_device(local_rank) call placed before any CUDA/HIP allocation. Once fixed, plain

BF16 training (no quantization) ran stably — the QLoRA detour had been solving a problem that no

longer existed once each rank owned its own GPU.

Attempt 2, second sub-issue — throughput: even after the fixes above, eager attention (no

Flash Attention 2 on this ROCm build) kept per-step latency around 120–150 s at 8,192-token

context. Reducing max_seq_len to 4,096 (still comfortably above the measured p99 of 3,628 tokens

across the dataset) roughly halved the attention cost without truncating any real example. Even

so, throughput meant the full 3-epoch run (6,060 steps) could not fit inside the cluster's 38-hour

per-job time limit. The final run was executed as a chain of 6 SLURM jobs

(afterany dependencies, each resuming from the latest checkpoint), spanning **2026-08-01 to

2026-08-10 (≈8 days 17 hours of wall-clock, cumulative queue + compute time)** before the final

job completed cleanly within its time limit and wrote the adapter.

Evaluation

**Training-time metrics only — no independent/adversarial evaluation has been run yet against this

specific checkpoint.** Treat the numbers below as evidence the fine-tune converged in-distribution,

not as a claim about real-world planner quality; a held-out multi-domain benchmark against the

pre-fine-tune baseline is planned but not yet executed as of this card's writing.

| Metric | Value | Source |

|---|---|---|

| Final training loss | 0.00017 | last logged training step (step 6,060/6,060, epoch 3.0) |

| Held-out eval loss | 0.0235 | 1,300-sample held-out split, epoch ≈2.97 |

| Held-out mean token accuracy | 0.9911 | same held-out split |

| Held-out mean token accuracy (training split, rolling) | ~0.99 throughout the final epoch | training logs |

The gap between the very low final training loss and the higher (but still low) held-out eval loss

is consistent with normal LoRA-SFT convergence on a synthetic, template-heavy dataset, not

independently confirmed to be free of overfitting on out-of-distribution phrasing — see

limitations below.

Known limitations

• Tied to one specific system prompt. The model was trained against a fixed, versioned

~2,500-token system prompt with a specific 16-category taxonomy and 22-tool catalog. This is not

a hypothetical caveat — a taxonomy/prompt mismatch between training and serving is the

documented root cause of a separate, earlier deployment failure in this project. Do not deploy

this adapter behind a different system prompt without retraining or at least re-validating.

• No independent evaluation yet. All numbers above are training/held-out-split metrics from

the same synthetic data distribution the model was trained on. No adversarial, out-of-distribution,

or human-graded evaluation has been run against this checkpoint at the time of writing.

• Inherits teacher-model biases. The training data was generated by a large teacher model

decomposing queries into the target taxonomy; any systematic blind spots or stylistic biases in

the teacher's decompositions carry over.

• Narrow by design. This is not a chat model and will not behave like one — it is trained to

emit only a JSON array, nothing else, and has not been evaluated for any other task.

• Quantized variants trade precision for size. The Q4_K_M GGUF in this repo (4.90 bits/weight,

~4.7 GB) is the recommended deployment format for constrained VRAM; F16 GGUF and BF16 safetensors

are also published for full-precision use or further fine-tuning.

Files in this repository

| File | Format | Size | Use case |

|---|---|---|---|

| qwen3-planner-q4_k_m.gguf | GGUF, Q4_K_M (4.90 BPW) | ~4.7 GB | Recommended for local/edge inference (e.g. Ollama) |

| qwen3-planner-f16.gguf | GGUF, F16 | ~16 GB | Full-precision GGUF inference |

The merged BF16 safetensors checkpoint is published separately at

h3rb3rn/qwen3-planner-sft.

Acknowledgments

Training was performed on LUMI, a EuroHPC Joint Undertaking supercomputer hosted by CSC

(Finland) and the LUMI consortium, under EuroHPC allocation project_465003058. Base model and

tokenizer: Qwen3-8B (Apache 2.0).