---

license: apache-2.0

base_model: Qwen/Qwen3.5-4B

base_model_relation: quantized

language:

• en

library_name: gguf

pipeline_tag: text-generation

tags:

• hob-forge
• edition
• small-gpu
• gguf
• qwen3.5
• llama.cpp
• ollama
• lm-studio
• imatrix
• iq4_xs
• q4_k_m
• q5_k_m
• q6_k
• q8_0
• 8gb
• cpu
• laptop
• instruct
• conversational

---

Qwen3.5-4B-Instruct — Hob Forge Edition (GGUF)

**Qwen's excellent 4B, set up correctly for small GPUs and modest RAM — with everything

measured, nothing projected.* This is a Hob Forge edition*: we didn't train this model

(all credit to Qwen, Apache-2.0); we quantized it

with a current toolchain, verified the chat template and tool-calling actually work,

measured speed / memory / quality-loss for every file, and wrote the run guide we wished

existed. If it's in a table below, we ran it on real hardware.

The headline you won't find on other GGUF pages: this is a hybrid-attention

architecture (8 full-attention layers + linear-attention DeltaNet + a 1-layer MTP block) —

its KV cache is ~5× smaller than a dense 4B. Measured: 8K context costs 256 MiB;

32K costs 1 GiB. Long context on tiny hardware is this model's superpower.

Which file? (measured on RTX 5070, -ngl 99, llama.cpp b368b24c)

| File | Size | Gen speed tg128 | Perplexity (wikitext-2, 120 chunks) | Note |

|---|---|---|---|---|

| Q4_K_M ⭐ | 2.58 GiB | 138 t/s | 10.030 ±0.157 | recommended default (imatrix) |

| Q5_K_M | 2.93 GiB | 131 t/s | 10.009 ±0.158 | quality step up |

| Q6_K | 3.31 GiB | 120 t/s | 9.887 ±0.155 | near-lossless |

| Q8_0 | 4.28 GiB | 102 t/s | 9.832 ±0.154 | reference quality |

| IQ4_XS | 2.39 GiB | 45 t/s ⚠ | 10.094 ±0.159 | smallest, but i-quant dequant is ~3× slower on RTX cards — only pick this if the last 200MB matters more than speed |

| F16 | 8.42 GiB | — | 9.871 ±0.155 | conversion source |

F16 baseline shows the whole ladder loses almost nothing: Q4_K_M sits +0.16 PPL from F16,

inside ~1σ. imatrix (for Q4_K_M and IQ4_XS) computed from a 2MB wikitext-2-train slice,

250 chunks — the calibration file ships in this repo (calib/).

Memory budget (measured, not projected)

Architecture-fixed overheads (identical for every quant): KV cache + 50 MiB recurrent-state

• ~70–96 MiB compute buffer.

| Context | KV cache | Total @ Q4_K_M | Total @ Q8_0 | Fits 4GB? | Fits 8GB? |

|---|---|---|---|---|---|

| 4K | 128 MiB | ~2.8 GiB | ~4.5 GiB | ✅ Q4 | ✅ |

| 8K | 256 MiB | ~2.9 GiB | ~4.7 GiB | ✅ Q4 | ✅ |

| 16K | 512 MiB | ~3.2 GiB | ~4.9 GiB | ✅ Q4 | ✅ |

| 32K | 1 GiB | ~3.7 GiB | ~5.4 GiB | ⚠ tight | ✅ |

Yes — 32K context, whole model, under 4GB at Q4_K_M. Every number from llama.cpp's

own allocator logs on our hardware.

Run it

# ollama
ollama run hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

# llama.cpp (recent build required — this arch needs 2026 support)
llama-cli -m Qwen3.5-4B-Instruct-Q4_K_M.gguf -st -p "your prompt" -ngl 99 -c 8192

Sampling (Qwen's recommendations, verified here): thinking mode `temp 0.6, top_p 0.95,

top_k 20; non-thinking temp 0.7, top_p 0.8, top_k 20`. This model thinks by default —

it emits reasoning before answering. Turn it off: think:false (top-level, ollama API) /

enable_thinking=False (transformers) / strip <think> blocks client-side for llama.cpp.

Full walkthrough — zero-to-first-tool-call on an 8GB-class GPU and a standard desktop,

with the troubleshooting we earned building this — in RUNNING.md.

Provenance & method

• Base: Qwen/Qwen3.5-4B (Apache-2.0) — untouched weights, full multimodal snapshot converted text-only.
• Toolchain: llama.cpp convert (build 2026-08-15) + quantize (build 2026-08-12). Chat template

verified by rendering tests (system/no-system/tools × both); tool-calling exercised with a real call.

• Evals here measure our files (quantization quality), not the model's intelligence —

for capability benchmarks see Qwen's card. PPL runs used identical chunks across all quants.

• No training data involved; nothing to decontaminate. No abliteration anywhere in lineage.

Limits

A 4B is a 4B: strong for its size at chat, coding assistance, and tool use; not a frontier

model. IQ4_XS speed caveat above. Vision components of the base are not included (text-only

GGUFs). MTP block included in F16 but speculative decoding needs runtime support.

---

*Hob Forge — the small-GPU and modest-RAM champion. Measured budgets, honest tables, real

support. If something in this card doesn't reproduce on your machine, open a discussion —

we answer.*