---

license: apache-2.0

base_model: DeepHat/DeepHat-V1-7B

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

tags:

- gguf

- llama.cpp

- imatrix

- quantized

- qwen2

- code

- cybersecurity

- devops

language:

- en

---

<p align="center"><img src="banner.png" alt="DeepHat-V1-7B GGUF" width="100%"></p>

┌────────────────────────────────────────────────────────────────────┐
│  DeepHat-V1-7B · GGUF                                               │
│  ───────────────────────────────────────────────────────────────   │
│  base   DeepHat/DeepHat-V1-7B   arch  qwen2 (Qwen2.5-Coder-7B ft)   │
│  domain cybersecurity · devops · code                              │
│  ladder Q3_K_M → Q8_0 + IQ4_XS   imatrix  ✓ (code+general)          │
│  fidelity  Q8 KLD 0.0019 · top-1 99.8% vs BF16                      │
└────────────────────────────────────────────────────────────────────┘

DeepHat-V1-7B — GGUF quant ladder

A full, imatrix-calibrated GGUF ladder of DeepHat/DeepHat-V1-7B — a

Qwen2.5-Coder-7B fine-tune focused on cybersecurity, devops and code. This is a faithful re-quantization

(all credit for the model goes to the DeepHat team); what's added here is the imatrix ladder, an objective

KLD fidelity table vs the BF16 reference, and an honest evaluation.

> First-mover note: at pack time no GGUF of this model existed. Quants Q3_K_M → Q8_0 + IQ4_XS, each with

> an importance matrix so the low-bit tiers stay sharp.

✅ Recommended files

| Use case | File | Why |

|---|---|---|

| Best all-round (16 GB RAM) | deephat-v1-7b-Q4_K_M.gguf | safe default, 98.97% top-1 vs BF16 |

| Quality-first (24 GB+) | deephat-v1-7b-Q6_K.gguf | near-lossless, 99.55% top-1 |

| Smallest usable (8–12 GB) | deephat-v1-7b-Q3_K_M.gguf | still 98.0% top-1 |

| Compact + sharp | deephat-v1-7b-IQ4_XS.gguf | imatrix IQ, 4.2 GB |

| Archival / eval | deephat-v1-7b-Q8_0.gguf | effectively lossless (KLD 0.0019) |

📦 Files

| Quant | Bits (BPW) | Size |

|---|---|---|

| Q3_K_M | 3-bit K | 3.81 GB |

| IQ4_XS | 4-bit IQ | 4.22 GB |

| Q4_K_M | 4-bit K | 4.68 GB |

| Q5_K_M | 5-bit K | 5.44 GB |

| Q6_K | 6-bit K | 6.25 GB |

| Q8_0 | 8.50 | 8.10 GB |

📊 Metrics — fidelity vs BF16 reference

Every tier measured against the unquantized BF16 GGUF (KL-divergence + PPL ratio + Top-1 agreement, general-English eval, c=2048).

| Quant | PPL(Q)/PPL(bf16) | Max KLD | KLD p99 | Top-1 match |

|---|---|---|---|---|

| Q8_0 | 1.000004 | 0.0019 | 0.0005 | 99.82% |

| Q6_K | 1.000340 | 0.0340 | 0.0079 | 99.55% |

| Q5_K_M | 1.000881 | 0.0624 | 0.018 | 99.50% |

| Q4_K_M | 0.998457 | 0.2078 | 0.030 | 98.97% |

| IQ4_XS | 0.996500 | 0.4858 | 0.041 | 98.74% |

| Q3_K_M | 1.002918 | 0.9454 | 0.109 | 98.02% |

The quantization is clean top-to-bottom — Q8 is effectively lossless and even Q3_K_M keeps 98% argmax agreement with BF16.

🧮 Will it fit?

| RAM / VRAM | Comfortable pick |

|---|---|

| 8 GB | Q3_K_M (short context) |

| 12 GB | Q4_K_M |

| 16 GB | Q5_K_M / Q6_K |

| 24 GB+ | Q8_0, long context |

7B at Q4–Q6 runs fast on a laptop GPU (Apple Silicon / consumer NVIDIA) or CPU.

🚀 How to run it

llama.cpp

llama-cli -m deephat-v1-7b-Q4_K_M.gguf -ngl 99 -c 8192 \
  -p "<|im_start|>user\nExplain how a SQL injection works and how to prevent it.<|im_end|>\n<|im_start|>assistant\n"

Ollama (a ready Modelfile with configurable context ships in this repo):

ollama create deephat -f deephat-8192ctx.Modelfile
ollama run deephat "Write a bash script that scans a subnet for open port 22."

Sampling: ChatML template, temperature 0.1–0.7, top_p 0.9, top_k 20, stop <|im_end|>.

⚠️ Good to know

• Chat template = upstream original (tool-calling preserved). These GGUFs embed DeepHat's original

Qwen tool-calling template (the XLAM/Qwen tools format), so native function-calling works just like the

source model. That template uses the Jinja tojson filter, which **older ollama / llama.cpp builds cannot

parse* (they error "Unknown (built-in) filter 'tojson'"*). If you hit that: update your runtime (recent

ollama/llama.cpp handle it), or override the template with the plain-ChatML scripts/chatml.jinja shipped here

(--chat-template in llama.cpp) — that variant loads everywhere but drops native tool-calling.

• Domain model. DeepHat is tuned for security / devops / code, not general chit-chat or general software

engineering. See the eval below.

🧪 Evaluation methodology

• Fidelity gate (passed): the KLD/PPL/Top-1 table above — every tier is faithful to BF16 (Q8 lossless).
• Agentic probe (reported honestly): swe-mix — 30 SWE-bench Verified instances (6 continuity anchors +

24 discriminating, 10 repos; spec in swe_mix.json), run through Claude Code + agent-bridge.js → Ollama

serving Q6_K, tool-call format openhands, temperature 0.1, in Docker.

Result: resolved = 0 / 30. The model did emit tool calls on 22/30 instances but ran very short episodes

(2–8 messages) — it doesn't sustain the long multi-step agentic loop a SWE-bench solve needs.

Honest caveat: (1) this is a 7B — small models routinely score 0–2 on SWE-bench Verified in a local

agentic harness; (2) SWE-bench is general-repo software engineering (Django/astropy/sympy), which is **not

DeepHat's domain (cybersecurity/devops). This number measures general agentic SWE ability, not** the model's

security/code strengths, and it is not comparable 1:1 to the official leaderboard.

Date: 2026-07-12. Small/hard probe, relative signal only.

This repo is fidelity-gated (like a non-general-SWE model release): the quant quality is proven; the SWE number

is published transparently rather than hidden.

🔁 Provenance & reproducibility

• scripts/reproduce.sh — exact convert → template-fix → imatrix → quantize commands.
• scripts/chatml.jinja — the clean chat template that replaced the upstream tojson one.
• imatrix corpus: ~793 KB, general text + Python/Rust/C code (386 chunks, c=512).
• reports/artifact-sha256sums.txt — SHA-256 of every GGUF.
• metrics/quant-summary-with-kld.{json,csv} — the table above, machine-readable.
• swe_mix.json — the 30-instance probe spec.

🗒️ Changelog

• 2026-07-12 — initial release. Ladder Q3_K_M→Q8_0 + IQ4_XS, imatrix, KLD sweep, cleaned chat template, honest SWE-mix eval.

Credit

Model: DeepHat/DeepHat-V1-7B (Apache-2.0). This repo only provides GGUF conversions + fidelity measurements. All model capability is the DeepHat team's work.