---

license: apache-2.0

base_model: OliviaRossi/gemma-4-12B-EsperGrug

language:

- en

tags:

- gguf

- llama.cpp

- quantization

- gemma-4

- reasoning

- coding

- agentic

- devops

pipeline_tag: text-generation

---

Gemma-4-12B-EsperGrug-GGUF

This repository contains official GGUF quantized releases of OliviaRossi/gemma-4-12B-EsperGrug, an exquisite hybrid model combining the dense, compact deductive reasoning of kai-os/Grug-12B with the agentic software engineering and DevOps execution capabilities of ValiantLabs/gemma-4-12B-it-Esper4.

These files are optimized for fast inference with llama.cpp, Ollama, LM Studio, Jan, and text-generation-webui on consumer GPUs, Apple Silicon, and CPU-only setups.

---

Ecosystem & Companion Repositories

• Original Unquantized Weights (Safetensors):

For raw BF16 weights, fine-tuning, or standard Hugging Face transformers deployments:

👉 OliviaRossi/gemma-4-12B-EsperGrug

• Recommended Chat Template & Prompting Harness:

To fully unlock this model's agentic tool-use, multi-turn reasoning traces, and structured code synthesis, apply the Tri-Craft harness:

👉 OliviaRossi/Gemma-4-Chat-Template-Tri-Craft-Harness

---

Quantization Details & File Breakdown

Gemma-4-12B features a massive 262,144-token vocabulary. In standard low-bit quantizations, converting vocabulary matrices aggressively can degrade rare tokens, programming keywords, and strict syntax parsing.

The primary build in this repository is Q5_K_L, which preserves full fidelity across code and reasoning tasks by applying:

• Body Layers (Attention & MLP): Q5_K_M quantization.
• Token Embeddings & Output LM Head: Kept at Q8_0 (--token-embedding-type q8_0 --output-tensor-type q8_0), eliminating vocabulary perplexity loss.

| Filename | Quant Type | Size | VRAM / RAM Required | Quality / Recommended Use Case |

| :--- | :--- | :--- | :--- | :--- |

| gemma-4-12B-EsperGrug-Q5_K_L.gguf | Q5_K_L | ~9.1 GB | ~11.5 GB | Flagship / Recommended. Q5_K_M body + Q8_0 embeddings. Near-lossless reasoning, zero token corruption. |

---

Usage Instructions

1. High-Performance Local Server (llama.cpp)

Launch an OpenAI-compatible API endpoint with full GPU offloading:

# Offload all 48 layers to GPU with an 8k context window
llama-server \
  -m gemma-4-12B-EsperGrug-Q5_K_L.gguf \
  -c 8192 \
  -ngl 99 \
  --port 8080 \
  --host 0.0.0.0

2. Interactive CLI

llama-cli \
  -m gemma-4-12B-EsperGrug-Q5_K_L.gguf \
  -c 4096 \
  -ngl 99 \
  --temp 0.2 \
  --top-p 0.95 \
  --repeat-penalty 1.05 \
  -p "Explain how to structure an asynchronous event loop in Rust and implement a cancellation token."

3. Using with Ollama

Create a custom model file:

# Modelfile
FROM ./gemma-4-12B-EsperGrug-Q5_K_L.gguf

PARAMETER temperature 0.2
PARAMETER top_p 0.95
PARAMETER repeat_penalty 1.05
PARAMETER stop "<end_of_turn>"
PARAMETER stop "<eos>"

TEMPLATE """{{ if .System }}<start_of_turn>system
{{ .System }}<end_of_turn>
{{ end }}{{ range .Messages }}<start_of_turn>{{ .Role }}
{{ .Content }}<end_of_turn>
{{ end }}<start_of_turn>model
"""

Build and run:

ollama create gemma4-espergrug -f Modelfile
ollama run gemma4-espergrug

4. Python Inference (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="./gemma-4-12B-EsperGrug-Q5_K_L.gguf",
    n_ctx=8192,
    n_gpu_layers=-1, # Offload all layers to GPU
    verbose=False
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are an expert systems engineer and reasoning specialist."},
        {"role": "user", "content": "Diagnose memory fragmentation issues in long-running Python worker pools."}
    ],
    temperature=0.2,
    max_tokens=1024
)

print(response["choices"][0]["message"]["content"])

---

Integrating the Recommended Tri-Craft Chat Template

For advanced agentic prompting and specialized system instructions, combine this GGUF model with the formatting templates from OliviaRossi/Gemma-4-Chat-Template-Tri-Craft-Harness:

# Pass the customized Jinja chat template directly into llama-server
llama-server \
  -m gemma-4-12B-EsperGrug-Q5_K_L.gguf \
  --chat-template-file ./tri_craft_chat_template.jinja \
  -c 8192 \
  -ngl 99

---

Recommended Generation Parameters

• Temperature: 0.2 (Low temperature preserves Grug's rigorous constraint checking and Esper4's syntactic code correctness)
• Top-P: 0.95
• Repetition Penalty: 1.05
• Context Window: 8,192 – 32,768 tokens (Native sliding-window attention supported)

---

Attribution & Acknowledgments

• Original Merge Creator: Olivia Rossi
• Base Architecture: Google Gemma-4
• Upstream Fine-Tuning:

* ValiantLabs/gemma-4-12B-it-Esper4 (Agentic synthesis & DevOps)

* kai-os/Grug-12B (Compact deduction & invariant checking)

• Quantization Tooling: llama.cpp by Georgi Gerganov and contributors