kikocis/gemma-4-31b-it-iq2_m-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "gemma",
    "base_model": "google/gemma-4-31b-it",
    "base_model_relation": "quantized",
    "library_name": "gguf",
    "pipeline_tag": "text-generation",
    "language": [
      "en"
    ],
    "tags": [
      "gguf",
      "quantized",
      "imatrix",
      "IQ2_M",
      "2-bit",
      "llama-cpp",
      "gemma4",
      "cli",
      "conversational",
      "text-generation"
    ],
    "frontmatter": {
      "license": "gemma",
      "base_model": "google/gemma-4-31b-it",
      "base_model_relation": "quantized",
      "library_name": "gguf",
      "pipeline_tag": "text-generation",
      "language": "[en]",
      "tags": [
        "gguf",
        "quantized",
        "imatrix",
        "IQ2_M",
        "2-bit",
        "llama-cpp",
        "gemma4",
        "cli",
        "conversational",
        "text-generation"
      ]
    },
    "hero_image_url": "",
    "summary": "Custom IQ2_M quantization of google/gemma-4-31b-it with an importance matrix calibrated on CLI / shell-assistant traces.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: gemma\nbase_model: google/gemma-4-31b-it\nbase_model_relation: quantized\nlibrary_name: gguf\npipeline_tag: text-generation\nlanguage: [en]\ntags:\n  - gguf\n  - quantized\n  - imatrix\n  - IQ2_M\n  - 2-bit\n  - llama-cpp\n  - gemma4\n  - cli\n  - conversational\n  - text-generation\n---\n\n# Gemma 4 31B IT — IQ2_M (CLI-focused imatrix)\n\nCustom IQ2_M quantization of [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it)\nwith an importance matrix calibrated on CLI / shell-assistant traces.\n\n- **Size**: 10.17 GB (6× smaller than f16)\n- **BPW**: 2.84\n- **Layers**: 60 (unmodified)\n- **Calibration**: imatrix from 4 chunks of CLI/bash-focused text\n- **Tool**: `llama.cpp` (`llama-quantize --imatrix ...`)\n\n## NL2Bash benchmark (50 examples, Stanford/Tellina test split)\n\n| Metric  | This IQ2_M | f16 baseline | Unsloth UD-IQ2_M |\n|---------|-----------:|-------------:|-----------------:|\n| Size    | **10.17 GB** | 61.4 GB | 10.75 GB |\n| Char-F1 | **84.71%** | 84.76% | 84.02% |\n| BLEU-1  | **44.72** | 43.94 | 42.38 |\n| BLEU-2  | **34.36** | 33.84 | 31.73 |\n| BLEU-4  | **22.39** | 21.02 | 18.64 |\n| Exact   | **12%** | 12% | 10% |\n\n**This IQ2_M matches f16 Char-F1 and beats f16 on BLEU-4** (22.39 vs 21.02)\nat 6× smaller. Also beats Unsloth UD-IQ2_M on every metric at 0.6 GB smaller.\n\nRun with: `llama-cli -m gemma4-31b-IQ2_M.gguf -ngl 99 --ctx-size 4096 --temp 0.1`\n\n\n## Cross-model benchmark comparison (first public eval)\n\n| Model | Size | Active | HumanEval+ | MBPP+ | BFCL v3 | NL2Bash F1 |\n|-------|------|--------|------------|-------|---------|------------|\n| **This repo (Gemma 4 31B IQ2_M)** | **10.4 GB** | 31B | **88.41%** | **82.01%** | 92.25% | **84.71%** |\n| Qwen3.6 Q8_0 (≈f16) | 35.2 GB | 3B | 81.10% | 82.80% | **95.25%** | — |\n| [Qwen3.6 IQ2_M (sibling)](https://huggingface.co/KikoCis/Qwen3.6-35B-A3B-IQ2_M-GGUF) | 11.1 GB | 3B | 80.49% | 78.31% | 94.75% | 81.63% |\n| Gemma 4 E4B Q8_0 | 7.8 GB | 4.5B | 73.78% | 73.28% | 93.75% | 79.75% |\n\n**This model is the best available for code generation in the 10-12 GB tier** — HumanEval+ 88.41% beats every Qwen3.6 variant including full precision.\n\n## How this compares\n\nFull comparison (8 quantizations, layer-importance study, ablation charts) at\n[otter.utopiaia.com](https://kikocisbot.github.io/gemma4-31b-study-site/) and\n[github.com/KikoCisBot/gemma4-31b-study](https://github.com/KikoCisBot/gemma4-31b-study).\n\nAt ~2.7 BPW, standard Q2_K collapses (F1 58.6%). Adaptive IQ2_M with a\nCLI-tuned imatrix holds **84.7% Char-F1** — functionally equivalent to f16 for\nshell-command generation.\n\n## Quickstart\n\n```bash\n# Download\nhuggingface-cli download KikoCis/gemma-4-31b-it-IQ2_M-GGUF gemma4-31b-IQ2_M.gguf --local-dir .\n\n# Run\nllama-cli -m gemma4-31b-IQ2_M.gguf -ngl 99 --ctx-size 4096 \\\n  --temp 0.1 -p \"List all processes using port 8080\"\n```\n\n## Files\n\n- `gemma4-31b-IQ2_M.gguf` — quantized weights\n- `gemma31b-imatrix.dat` — importance matrix used for calibration\n\n## Citation\n\nIf you find this useful, consider starring the\n[otter](https://github.com/KikoCisBot/gemma4-31b-study) repo.\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "quantized",
    "imatrix",
    "IQ2_M",
    "2-bit",
    "llama-cpp",
    "gemma4",
    "cli",
    "conversational",
    "text-generation",
    "en",
    "license:gemma",
    "endpoints_compatible",
    "region:us"
  ],
  "likes": 0,
  "downloads": 473,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-17T13:33:33.000Z",
  "created_at": "2026-04-15T12:52:46.000Z",
  "pipeline_tag": "text-generation",
  "library_name": "gguf"
}

{
  "_id": "69df8a1ec1d66e2b808687ab",
  "id": "KikoCis/gemma-4-31b-it-IQ2_M-GGUF",
  "modelId": "KikoCis/gemma-4-31b-it-IQ2_M-GGUF",
  "sha": "1097785bdae5cd60c862a55b794e10116848c90d",
  "createdAt": "2026-04-15T12:52:46.000Z",
  "lastModified": "2026-04-17T13:33:33.000Z",
  "author": "KikoCis",
  "downloads": 473,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "gguf",
  "siblings_count": 4
}