pearsonkyle/Ornith-1.0-9B-imatrix-GGUF overview

---

library_name: gguf

base_model:

• deepreinforce-ai/Ornith-1.0-9B

tags:

• gguf
• llama.cpp
• text-generation
• text-generation-inference
• transformers
• quantization
• quantized
• imatrix
• importance-matrix
• awq
• vision
• multimodal
• mmproj
• 2-bit
• 4-bit
• 5-bit
• iq2_m
• iq4_xs
• q5_k_m
• qwen3.5
• coder
• agentic
• reasoning
• tool-use
• function-calling

license: mit

language:

• en

pipeline_tag: text-generation

---

<div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; border: 1px solid #99f6e4; border-radius: 16px; box-shadow: 0 10px 15px -3px rgba(0,0,0,0.05), 0 4px 6px -2px rgba(0,0,0,0.05); overflow: hidden; background: #ffffff; margin-bottom: 30px;">

<div style="background: linear-gradient(135deg, #0d9488 0%, #134e4a 100%); padding: 24px; color: white;">

<div style="display: flex; align-items: center; justify-content: space-between; flex-wrap: wrap; gap: 10px;">

<h1 style="margin: 0; font-size: 26px; font-weight: 800; display: flex; align-items: center; gap: 12px; color: white; border: none;">🧊 deepreinforce-ai/Ornith-1.0-9B</h1>

<span style="background: #f59e0b; color: #1c1917; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; text-transform: uppercase; letter-spacing: 0.5px;">imatrix + AWQ + vision</span>

</div>

</div>

<div style="display: flex; gap: 8px; flex-wrap: wrap; padding: 12px 24px; background: #f8fafc; border-bottom: 1px solid #e2e8f0;">

<span style="background: #dbeafe; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #bfdbfe;">📦 IQ2_M / IQ4_XS / Q5_K_M · 3.4–6.0 GiB</span>

<span style="background: #d1fae5; color: #065f46; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #a7f3d0;">🥇 Q5_K_M near-lossless · KLD 0.002 · top_p 96%</span>

<span style="background: #fef3c7; color: #92400e; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #fde68a;">🧪 2-bit AWQ tool-call rescue · param-acc 5%→33%</span>

<span style="background: #ede9fe; color: #5b21b6; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #ddd6fe;">👁️ vision mmproj (Q8) included</span>

<span style="background: #fce7f3; color: #9d174d; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #fbcfe8;">🧠 reasoning · agentic coding</span>

</div>

<div style="padding: 24px; display: flex; flex-direction: column; gap: 20px;">

<div style="background: #f0fdfa; border-left: 5px solid #0d9488; padding: 16px; border-radius: 0 8px 8px 0;">

<h3 style="margin: 0 0 8px 0; font-size: 15px; color: #115e59; font-weight: 700; display: flex; align-items: center; gap: 6px;"><span>🧊</span> What this is</h3>

<p style="margin: 0; font-size: 13px; color: #334155; line-height: 1.7;">a <b>2 → 5-bit GGUF ladder</b> for <b>deepreinforce-ai/Ornith-1.0-9B</b> — a Qwen3.5-VL–based <b>agentic-coding reasoning model</b> (tool-calling, <code>&lt;think&gt;</code> chain-of-thought). Each quant is calibrated from real agentic-coding usage logs + wiki text, and <b>the calibration method is chosen per bit-width by agentic tool-call fidelity, not just KLD/PPL</b>: the <b>2-bit ships an AWQ build</b> (activation-aware scaling rescues tool-argument precision that plain 2-bit imatrix loses — param-acc 5%→33%), while 4/5-bit ship plain hybrid imatrix. <code>Q5_K_M</code> is near-lossless (KLD 0.002); <code>IQ4_XS</code> matches it agentically at 20% smaller. The <b>vision tower ships separately as a Q8 mmproj</b>. Plain GGUF, no custom runtime.</p>

</div>

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); gap: 15px; margin-top: 10px;">

<div style="border: 1px solid #e2e8f0; padding: 14px; border-radius: 8px; background: #fafafa; box-shadow: inset 0 2px 4px rgba(0,0,0,0.02);"><span style="font-weight: 700; color: #115e59; font-size: 12px; display: block; margin-bottom: 6px; text-transform: uppercase; letter-spacing: 0.5px;">📉 2.8× smaller on disk</span><span style="font-size: 13px; color: #4b5563; line-height: 1.5;">6.0 GiB on disk vs 16.7 GiB for the FP16 text trunk. Tuned for English + Python agentic-coding workloads (see calibration scope below).</span></div>

<div style="border: 1px solid #e2e8f0; padding: 14px; border-radius: 8px; background: #fafafa; box-shadow: inset 0 2px 4px rgba(0,0,0,0.02);"><span style="font-weight: 700; color: #115e59; font-size: 12px; display: block; margin-bottom: 6px; text-transform: uppercase; letter-spacing: 0.5px;">👁️ vision retained</span><span style="font-size: 13px; color: #4b5563; line-height: 1.5;">A 595 MiB Q8 <code>mmproj</code> (SigLIP-style vision encoder + projector) lets the quantized trunk still accept images via <code>--mmproj</code>.</span></div>

</div>

</div>

</div>

🧰 1. Files & comparison

Three quants spanning 2 → 5 bit, one variant per bit-width (so ollama run …:IQ2_M / :IQ4_XS / :Q5_K_M all resolve). The 2-bit ships an AWQ build, the 4/5-bit ship plain hybrid imatrix — see the agentic table below for why. FP16 text-trunk reference: 16.69 GiB (not included; fetch from deepreinforce-ai/Ornith-1.0-9B). Vision ships separately as a Q8 mmproj (595 MiB).

| File | Method | Size (GiB) | BPW | PPL (gen) | KLD med (gen) | top_p (gen) | KLD med (tools) | top_p (tools) |

|---|---|---:|---:|---:|---:|---:|---:|---:|

| FP16 (reference) | — | 16.69 | 16.000 | 5.8874 | 0.0000 | 100.0% | 0.0000 | 100.0% |

| IQ2_M | AWQ + imatrix | 3.36 | 3.223 | 6.7108 | 0.1163 | 79.73% | 0.3315 | 61.09% |

| IQ4_XS | hybrid imatrix | 4.84 | 4.643 | 5.9446 | 0.0096 | 93.08% | 0.0351 | 74.23% |

| Q5_K_M | hybrid imatrix | 6.02 | 5.779 | 5.9759 | 0.0021 | 96.41% | 0.0083 | 82.45% |

> IQ4_XS matches the near-lossless Q5_K_M on agentic tool-calling (below) at 20% smaller — a good default. IQ2_M is the smallest quant that still tool-calls usefully.

🧪 Agentic tool-call fidelity — and why the 2-bit is AWQ

Static KLD/PPL measures how closely the quant's logits track FP16; it does not measure whether the model still uses tools correctly. We replay 25 held-out real agentic-coding sessions (disjoint from calibration) through llama-server, scoring per assistant turn whether the model selects the right tool (tool-sel) and fills the right arguments (param-acc). 3 reps, mean ± σ.

| Quant (3.22 / 4.64 / 5.78 bpw) | tool-sel | param-acc | schema-valid |

|---|---:|---:|---:|

| IQ2_M — imatrix (rejected) | 0.31 ± .08 | 0.05 ± .02 | 0.85 |

| IQ2_M — AWQ (shipped) | 0.54 ± .03 | 0.33 ± .03 | 0.93 |

| IQ4_XS — imatrix (shipped) | 0.65 ± .02 | 0.36 ± .03 | 0.98 |

| Q5_K_M — imatrix (shipped) | 0.64 ± .02 | 0.35 ± .02 | 0.96 |

At 2 bits the plain-imatrix quant is agentically broken: it emits schema-valid tool calls but fills the wrong arguments (5% param-acc) — even though its static tools-KLD (0.30) is better than the AWQ build's (0.33). AWQ's activation-aware channel scaling rescues argument precision at 2-bit (5% → 33% param-acc, +23 pts tool-sel), so the shipped IQ2_M is the AWQ build. At 4-bit the effect reverses (imatrix leads tool-sel 0.65 vs 0.58 and AWQ's RMSNorm fold slightly perturbs general text), so IQ4_XS/Q5_K_M ship plain imatrix. Takeaway: choose the calibration method by the metric you care about — static fidelity ≠ agentic capability.

> Sampling: T=0.25, top_p=0.95, top_k=20, reasoning disabled for eval throughput — absolute scores understate this <think> model, but the relative ranking (the point here) is stable across reps. Harness: Quant-Tuner run_toolcall_reps.py.

<div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; border: 1px solid #bfdbfe; border-radius: 12px; background: #eff6ff; padding: 16px; margin: 16px 0; color: #1e3a8a; font-size: 13px; line-height: 1.7;">

<b>📋 Calibration scope — English &amp; Python, agentic coding.</b> The importance matrix (and the windowed packing that shaped it) was calibrated on <b>real agentic-coding sessions that are overwhelmingly English-language and Python-centric</b>, captured from <b>Claude Code, opencode, and qwen code</b> (tool-turn share ≈ 90%). At low bit-widths the codebook's precision is spent where those logs put it: English prompts and Python-flavored tool use (read / edit / bash / grep / write, etc.). Expect <b>weaker fidelity on other natural languages, non-Python ecosystems, and general-chat / vision-heavy workloads</b>.

</div>

🤖 SWE-rebench (agentic coding, Q5_K_M)

The Q5_K_M trunk is evaluated on 10 real-world coding issues from nebius/SWE-rebench using the OpenAI Agents SDK pointed at a local llama-server. For each issue the agent gets the problem statement and a live bash tool that shells into a dedicated Docker container with the repo checked out at the failing commit; it reads files, runs tests, and edits code until it produces a git diff or hits the step cap. The patch is graded by actually running the repo's FAIL_TO_PASS suite inside the container — pass/fail is real execution, not fuzzy matching. (Full-loop SWE-rebench on the smaller quants is expensive; the §1 tool-call replay is the lighter agentic proxy used to rank them.)

| Metric | Q5_K_M |

|:-|:-|

| Issues | 10 |

| Patch Rate | 100% (10/10) |

| Pass Rate | 30% (3/10) |

| Max Turns / Timeout | 0% |

| Mean Steps | 20.0 |

| Mean Tokens | 192K |

| Tool Error Rate | 13.9% |

| Mean Wall | 399s |

> Sampling: temperature=0.25, top_p=0.95, top_k=20, max_tokens=32768, ctx=131072, thinking=true. Tested on Apple Silicon (Metal) with the SWE-rebench linux/amd64 images under Docker emulation.

---

🔬 2. How they were made

<div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; display: flex; flex-direction: column; gap: 16px; margin-bottom: 24px;">

<div style="border: 1px solid #99f6e4; border-radius: 12px; overflow: hidden; background: #ffffff;">

<div style="background: linear-gradient(135deg, #0d9488 0%, #115e59 100%); padding: 12px 16px; color: white; font-weight: 700; font-size: 14px;">🧮 2.1 Hybrid importance matrix</div>

<div style="padding: 16px; font-size: 13px; color: #334155; line-height: 1.7;">

<p style="margin: 0 0 10px 0;">At low bit-widths the quantizer must decide <i>where</i> to spend its limited precision. An <b>importance matrix</b> measures, per input channel, how much that channel drives each layer's output on a calibration corpus, and tells <code>llama-quantize</code> to preserve the high-impact channels. This release uses a <b>hybrid</b> imatrix blending activation energy <code>E[a²]</code> with weight-column energy <code>‖W[:, c]²‖ · E[a²]</code>, collected at ctx=4096. Linear-attention / SSM tensors (Ornith is a Qwen3.5 hybrid architecture) pass through with raw <code>E[a²]</code>. This is what <b>IQ4_XS</b> and <b>Q5_K_M</b> use. The output is a standard GGUF with no runtime overhead.</p>

</div>

</div>

<div style="border: 1px solid #fde68a; border-radius: 12px; overflow: hidden; background: #ffffff;">

<div style="background: linear-gradient(135deg, #f59e0b 0%, #b45309 100%); padding: 12px 16px; color: white; font-weight: 700; font-size: 14px;">🧪 2.1b AWQ for the 2-bit (activation-aware scaling)</div>

<div style="padding: 16px; font-size: 13px; color: #334155; line-height: 1.7;">

<p style="margin: 0 0 10px 0;">At 2 bits, plain imatrix keeps the model's <i>logits</i> close to FP16 but loses the precision to fill correct <b>tool-call arguments</b> (see §1's agentic table: 5% param-acc). <b>AWQ</b> (activation-aware weight quantization) first rescales each input channel by a per-channel factor <code>s_c = mean(|x_c|)^α</code> and folds the inverse into the preceding RMSNorm — mathematically identity in FP16, but it redistributes which channels land on the finer quant levels. The α grid-search and the scale statistics are collected on the <b>same agentic-coding corpus</b>, strided across the whole file so tool-call turns are actually seen. On Ornith's hybrid trunk AWQ scales the full-attention layers' <code>q/k/v</code> + every layer's MLP; the linear-attention layers pass through. Net at 2-bit: <b>param-acc 5% → 33%, tool-sel +23 pts</b> — so <code>IQ2_M</code> ships the AWQ build. (At 4-bit+ the fold's perturbation outweighs the benefit, so those stay imatrix.)</p>

</div>

</div>

<div style="border: 1px solid #ddd6fe; border-radius: 12px; overflow: hidden; background: #ffffff;">

<div style="background: linear-gradient(135deg, #7c3aed 0%, #5b21b6 100%); padding: 12px 16px; color: white; font-weight: 700; font-size: 14px;">👁️ 2.2 Vision projector (mmproj)</div>

<div style="padding: 16px; font-size: 13px; color: #334155; line-height: 1.7;">

<p style="margin: 0 0 10px 0;">Ornith is a vision-language model. The text trunk is quantized here; its <b>SigLIP-style vision encoder + projector</b> is exported separately as <a href="./mmproj-Ornith-1.0-9B-Q8_0.gguf"><code>mmproj-Ornith-1.0-9B-Q8_0.gguf</code></a> (595 MiB, Q8, <code>qwen3vl_merger</code> projector, 27 vision blocks). Pass it with <code>--mmproj</code> alongside any trunk quant to feed images. The projector is small relative to the trunk, so it is kept near-lossless at Q8 while the trunk absorbs the aggressive quantization.</p>

</div>

</div>

<div style="border: 1px solid #bfdbfe; border-radius: 12px; overflow: hidden; background: #ffffff;">

<div style="background: linear-gradient(135deg, #2563eb 0%, #1e3a8a 100%); padding: 12px 16px; color: white; font-weight: 700; font-size: 14px;">📚 2.3 Calibration & evaluation data</div>

<div style="padding: 16px; font-size: 13px; color: #334155; line-height: 1.7;">

<p style="margin: 0 0 10px 0;">Calibration and every eval corpus are disjoint by construction — both the static <i>tools</i> KLD corpus and the <b>agentic tool-call replay</b> (§1) draw from the held-out 10% of sessions, never seen by calibration — so §1 measures generalization, not fit. All shipped under <code>calibration_data/</code>.</p>

<table style="width:100%; border-collapse:collapse; font-size:12px; margin:0;">

<thead><tr style="background:#eff6ff;"><th style="padding:8px 10px; border:1px solid #bfdbfe; text-align:left; color:#1e3a8a;">Corpus</th><th style="padding:8px 10px; border:1px solid #bfdbfe; text-align:left; color:#1e3a8a;">Source</th><th style="padding:8px 10px; border:1px solid #bfdbfe; text-align:left; color:#1e3a8a;">Used for</th></tr></thead>

<tbody>

<tr><td style="padding:8px 10px; border:1px solid #bfdbfe;"><b>Calibration</b></td><td style="padding:8px 10px; border:1px solid #bfdbfe;">~500k tokens of agentic-coding usage-log text (windowed, ≈90% tool-turn) + all of <code>wiki.test.raw</code></td><td style="padding:8px 10px; border:1px solid #bfdbfe;">hybrid imatrix collection</td></tr>

<tr><td style="padding:8px 10px; border:1px solid #bfdbfe;"><b>Eval — tools</b> (in-distribution)</td><td style="padding:8px 10px; border:1px solid #bfdbfe;">held-out logtrain session slice (10%), windowed like calibration but disjoint from it</td><td style="padding:8px 10px; border:1px solid #bfdbfe;"><b>§1 <i>tools</i> columns (KLD · top_p)</b></td></tr>

<tr><td style="padding:8px 10px; border:1px solid #bfdbfe;"><b>Eval — general</b></td><td style="padding:8px 10px; border:1px solid #bfdbfe;"><code>combined_en_tiny</code> (broad English) from the eaddario/imatrix-calibration dataset</td><td style="padding:8px 10px; border:1px solid #bfdbfe;"><b>§1 <i>general</i> columns (PPL · KLD · top_p)</b></td></tr>

</tbody>

</table>

</div>

</div>

</div>

---

🚀 3. Usage

Quick start with Ollama

Each quant is exposed as a tag (the filename's quant suffix):

ollama run hf.co/pearsonkyle/Ornith-1.0-9B-imatrix-GGUF:IQ2_M    # 3.4 GiB · AWQ · smallest that tool-calls
ollama run hf.co/pearsonkyle/Ornith-1.0-9B-imatrix-GGUF:IQ4_XS   # 4.8 GiB · imatrix · best size/quality
ollama run hf.co/pearsonkyle/Ornith-1.0-9B-imatrix-GGUF:Q5_K_M   # 6.0 GiB · imatrix · near-lossless

Building llama.cpp from source (GPU)

apt-get update && apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON   # -DGGML_CUDA=OFF for CPU/Metal
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-server llama-mtmd-cli
cp llama.cpp/build/bin/llama-* llama.cpp/

Running the server (text)

 ./llama-server \
    --model Ornith-1.0-9B-Q5_K_M.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 999 \
    --flash-attn on \
    --host 0.0.0.0 --port 1234

Running with vision (image input)

Pass the bundled Q8 projector with --mmproj:

 ./llama-server \
    --model Ornith-1.0-9B-Q5_K_M.gguf \
    --mmproj mmproj-Ornith-1.0-9B-Q8_0.gguf \
    --ctx-size 16384 --n-gpu-layers 999 --flash-attn on \
    --host 0.0.0.0 --port 1234
# or one-shot on an image:
 ./llama-mtmd-cli -m Ornith-1.0-9B-Q5_K_M.gguf --mmproj mmproj-Ornith-1.0-9B-Q8_0.gguf \
    --image photo.jpg -p "Describe this image."

Querying via the OpenAI-compatible API

import json, urllib.request

def ask(content, max_tokens=1024):
    body = {
        "messages": [{"role": "user", "content": content}],
        "max_tokens": max_tokens,
        # Ornith is a reasoning model: it emits a <think> chain-of-thought before
        # the answer. Give it room, or set enable_thinking False for terse replies.
        "chat_template_kwargs": {"enable_thinking": True},
    }
    req = urllib.request.Request("http://127.0.0.1:1234/v1/chat/completions",
                                 json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
    return json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"]

print(ask("Write a Python function that reverses a linked list."))

---

🪪 4. License & attribution

• Licensed MIT, inherited from the base model deepreinforce-ai/Ornith-1.0-9B. Confirm the current terms on the base model card before redistributing.
• Base weights: deepreinforce-ai/Ornith-1.0-9B — a Qwen3.5-VL–based agentic-coding reasoning model (post-trained on Qwen 3.5 + Gemma 4).
• Calibration + quantization performed locally with Quant-Tuner; vendored llama.cpp at commit f3e1828. The Q5_K_M quant was produced with a hybrid imatrix on the calibration corpora below.
• Calibration data (usage logs) scraped using LogMiner.