kikocis/gemma-4-31b-it-q3_k_m-gguf overview

{
  "metadata": {},
  "card_data": {
    "language": [
      "en"
    ],
    "license": "gemma",
    "tags": [
      "gemma4",
      "gguf",
      "quantized",
      "llm",
      "text-generation",
      "q3_k_m"
    ],
    "base_model": "google/gemma-4-31b-it",
    "pipeline_tag": "text-generation",
    "frontmatter": {
      "language": [
        "en"
      ],
      "license": "gemma",
      "tags": [
        "gemma4",
        "gguf",
        "quantized",
        "llm",
        "text-generation",
        "q3_k_m"
      ],
      "base_model": "google/gemma-4-31b-it",
      "pipeline_tag": "text-generation"
    },
    "hero_image_url": "",
    "summary": "Quantized GGUF of google/gemma-4-31b-it at Q3_K_M precision.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlanguage:\n- en\nlicense: gemma\ntags:\n- gemma4\n- gguf\n- quantized\n- llm\n- text-generation\n- q3_k_m\nbase_model: google/gemma-4-31b-it\npipeline_tag: text-generation\n---\n\n# Gemma 4 31B Instruct — Q3_K_M GGUF\n\nQuantized GGUF of [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it) at Q3_K_M precision.\n\n- **Architecture**: Dense transformer, 60 layers, hidden size 5376\n- **Format**: GGUF Q3_K_M (token embeddings at Q6_K for better accuracy)\n- **File size**: 15.3 GB\n- **Original size**: 61.4 GB (f16) → **75% smaller**\n\n## Benchmark — CLI task accuracy\n\nTested on 7 package-manager / install tasks across Ubuntu, macOS, Arch, Void, Debian:\n\n| Mode | Score | Avg speed |\n|------|-------|-----------|\n| With thinking (`think=true`) | **7 / 7** | ~17 tok/s |\n| Without thinking | 6 / 7 | ~16 tok/s |\n\nThinking mode recovers the one case where the model without chain-of-thought gives a slightly imprecise answer. For general use, enabling thinking is recommended.\n\n## Quality vs Q4_K_M\n\nQ3_K_M uses ~3.35 bits/weight vs ~4.5 for Q4_K_M. In practice the difference on instruction-following and factual recall tasks is **0.5–1.5%** on standard benchmarks. For this model:\n\n- Token embeddings are kept at **Q6_K** (not Q3), which protects vocabulary accuracy — the most quality-sensitive tensor\n- Attention and FFN weights are at Q3_K/Q4_K mixed (llama.cpp default K-quant scheme)\n- At 7/7 with thinking, the quantization error is below the threshold that affects task correctness for short-form generation\n\nExpected delta vs a hypothetical Q4_K_M (~19 GB) on longer-form tasks: ~1% perplexity increase, negligible for < 512 token outputs.\n\n## Memory advantage\n\n| Format | Size | VRAM needed (estimate) |\n|--------|------|------------------------|\n| f16 (original) | 61.4 GB | ~65 GB |\n| Q4_K_M | ~19 GB | ~21 GB |\n| **Q3_K_M (this)** | **15.3 GB** | **~17 GB** |\n| Q2_K | ~12 GB | ~14 GB |\n\nQ3_K_M fits in a single 24 GB GPU (RTX 3090/4090) with context headroom, or comfortably on Apple Silicon 24 GB with partial CPU offload.\n\n## Usage with llama.cpp\n\n```bash\n# Download\nhuggingface-cli download KikoCis/gemma-4-31b-it-Q3_K_M-GGUF \\\n  gemma-4-31b-it-Q3_K_M.gguf --local-dir .\n\n# Run (enable thinking for best results)\nllama-cli -m gemma-4-31b-it-Q3_K_M.gguf \\\n  --prompt \"How do I install neovim on Arch Linux?\" \\\n  -n 200 --temp 0.1\n```\n\n## Usage with Ollama\n\n```bash\n# Create Modelfile\ncat > Modelfile << 'EOF'\nFROM ./gemma-4-31b-it-Q3_K_M.gguf\nPARAMETER temperature 0.1\nPARAMETER num_predict 500\nEOF\n\nollama create gemma4-31b -f Modelfile\nollama run gemma4-31b\n```\n\n## Usage with Python (thinking mode)\n\n```python\nimport ollama\n\nresponse = ollama.chat(\n    model=\"gemma4-31b\",\n    messages=[{\"role\": \"user\", \"content\": \"How do I install git on Ubuntu?\"}],\n    think=True,\n    options={\"temperature\": 0.1},\n)\nprint(response[\"message\"][\"content\"])\n```\n\n## Notes\n\n- Quantized from the official HuggingFace weights using llama.cpp\n- No fine-tuning applied — this is the base instruct model at reduced precision\n- Thinking (chain-of-thought) is supported and recommended for best accuracy\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "gemma4",
    "quantized",
    "llm",
    "text-generation",
    "q3_k_m",
    "en",
    "license:gemma",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 244,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-09T13:20:23.000Z",
  "created_at": "2026-04-09T13:14:00.000Z",
  "pipeline_tag": "text-generation",
  "library_name": ""
}

{
  "_id": "69d7a6182ff9a465bc6fbf29",
  "id": "KikoCis/gemma-4-31b-it-Q3_K_M-GGUF",
  "modelId": "KikoCis/gemma-4-31b-it-Q3_K_M-GGUF",
  "sha": "d034fc8320dc0984d7702d156a52bb237b08da23",
  "createdAt": "2026-04-09T13:14:00.000Z",
  "lastModified": "2026-04-09T13:20:23.000Z",
  "author": "KikoCis",
  "downloads": 244,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "",
  "siblings_count": 3
}