kikocis/gemma-4-31b-it-iq3_xs-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "tags": [
      "3-bit",
      "IQ3_XS",
      "cli",
      "conversational",
      "gemma4",
      "gguf",
      "imatrix",
      "llama-cpp",
      "package-management",
      "quantized",
      "text-generation"
    ],
    "base_model": "google/gemma-4-31b-it",
    "model_type": "gemma4",
    "quantized_by": "KikoCis",
    "pipeline_tag": "text-generation",
    "language": [
      "en"
    ],
    "base_model_relation": "quantized",
    "library_name": "gguf",
    "frontmatter": {
      "license": "apache-2.0",
      "tags": [
        "3-bit",
        "IQ3_XS",
        "cli",
        "conversational",
        "gemma4",
        "gguf",
        "imatrix",
        "llama-cpp",
        "package-management",
        "quantized",
        "text-generation"
      ],
      "base_model": "google/gemma-4-31b-it",
      "model_type": "gemma4",
      "quantized_by": "KikoCis",
      "pipeline_tag": "text-generation",
      "language": [
        "en"
      ],
      "base_model_relation": "quantized",
      "library_name": "gguf"
    },
    "hero_image_url": "",
    "summary": "Gemma 4 31B Instruct quantized to **IQ3_XS** (3.40 BPW, 13.1GB) using llama.cpp with importance matrix calibration on CLI/package management data. **Matches f16 baseline on Char-F1 (84.76%) at 1/4.7 the size** (50-example NL2Bash benchmark). > ### 💡 Looking for the smallest possible variant? > Try our **gemma-4-31b-it-IQ2_M-GGUF** — 10.17 GB with F1 84.71% and **BLEU-4 22.39 (beats f16 at 6× smaller)**. Custom CLI-tuned imatrix on IQ2_M.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\ntags:\n- 3-bit\n- IQ3_XS\n- cli\n- conversational\n- gemma4\n- gguf\n- imatrix\n- llama-cpp\n- package-management\n- quantized\n- text-generation\nbase_model: google/gemma-4-31b-it\nmodel_type: gemma4\nquantized_by: KikoCis\npipeline_tag: text-generation\nlanguage:\n- en\nbase_model_relation: quantized\nlibrary_name: gguf\n---\n\n# Gemma 4 31B IT — IQ3_XS GGUF\n\nGemma 4 31B Instruct quantized to **IQ3_XS** (3.40 BPW, 13.1GB) using llama.cpp with importance matrix calibration on CLI/package management data.\n\n**Matches f16 baseline on Char-F1 (84.76%) at 1/4.7 the size** (50-example NL2Bash benchmark).\n\n> ### 💡 Looking for the smallest possible variant?\n> Try our **[gemma-4-31b-it-IQ2_M-GGUF](https://huggingface.co/KikoCis/gemma-4-31b-it-IQ2_M-GGUF)** — 10.17 GB with F1 84.71% and **BLEU-4 22.39 (beats f16 at 6× smaller)**. Custom CLI-tuned imatrix on IQ2_M.\n\n## Key Stats\n\n| Metric | Value |\n|--------|-------|\n| **Base model** | google/gemma-4-31b-it |\n| **Quantization** | IQ3_XS (3.40 BPW) |\n| **Size** | 13.1 GB |\n| **Layers** | 60 (full model, no pruning) |\n| **NL2Bash Char-F1** | 84.76% (= f16 baseline) |\n| **CLI 7/7** | 7/7 with thinking enabled |\n\n## NL2Bash Benchmark — Full Comparison\n\n50 examples from the official Stanford/Tellina test split, reasoning OFF, max_tokens=200, temp=0.1, sorted by Char-F1.\n\n| Model | Size | BPW | Char-F1 | BLEU-1 | BLEU-2 | BLEU-4 | EM |\n|-------|------|-----|---------|--------|--------|--------|-----|\n| [Unsloth UD-IQ3_XXS](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF) | 11.84 GB | ~2.98 | **85.06%** | **46.30** | **34.64** | 20.26 | 12% |\n| Base f16 (full precision) | 61.4 GB | 16.0 | 84.76% | 43.94 | 33.84 | 21.02 | 12% |\n| **Base IQ3_XS** (this model) | **13.1 GB** | 3.40 | 84.76% | 42.77 | 31.46 | 18.95 | 8% |\n| **✨ [Sibling IQ2_M (CLI imatrix)](https://huggingface.co/KikoCis/gemma-4-31b-it-IQ2_M-GGUF)** | **10.17 GB** | 2.84 | **84.71%** | 44.72 | 34.36 | **22.39** | **12%** |\n| [Unsloth UD-IQ2_M](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF) | 10.75 GB | ~2.70 | 84.02% | 42.38 | 31.73 | 18.64 | 10% |\n| [Unsloth UD-IQ2_XXS](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF) | 8.53 GB | ~2.06 | 76.82% | 33.03 | 22.17 | 11.49 | 0% |\n| Base Q2_K (3rd party) | 11.0 GB | 2.70 | 58.60% | 18.82 | 12.79 | 6.92 | 0% |\n\n### Key observations\n\n- **This IQ3_XS ties f16 on Char-F1** (84.76% vs 84.76%) and comes within 2 F1 of the best available quantization\n- **Unsloth Dynamic 2.0 is the Pareto frontier**: UD-IQ2_M (10.75 GB) is strictly better than plain Q2_K (+25.4 F1 points at same BPW) thanks to per-layer adaptive bit allocation and Gemma-4-specific imatrix\n- **Q2_K at 2.7 BPW collapses** on Gemma 4 (F1 58.6%, produces repetitive output). Standard 2-bit scalar quantization is not viable for this architecture\n- **f16 only wins on token-exact metrics** (BLEU-4, EM) — IQ3_XS produces semantically equivalent commands with minor stylistic variations (different flag order, quoting style)\n\n### Recommendation\n\n- **Smallest working quant**: Unsloth UD-IQ2_M (10.75 GB, F1 84%)\n- **Best overall**: Unsloth UD-IQ3_XXS (11.84 GB, F1 85%)\n- **This model (IQ3_XS, 13.1 GB)**: simpler imatrix calibration (CLI-focused), slightly larger but within 0.3 F1 of UD-IQ3_XXS\n\nFull per-question predictions + layer analysis study: https://huggingface.co/datasets/KikoCis/gemma4-31b-layer-study\n\n## CLI Benchmark Results (7/7 with thinking)\n\nTested with `llama-cli -cnv --reasoning on --reasoning-budget 512`:\n\n| Test | Result |\n|------|--------|\n| Install neofetch on Void Linux | `sudo xbps-install -S neofetch` |\n| Install htop on Ubuntu | `sudo apt install htop` |\n| Search ripgrep on Arch | `pacman -Ss ripgrep` |\n| Search packages on Void | `xbps-query -S <package_name>` |\n| Add cargo to PATH in zsh | `echo 'export PATH=\"$HOME/.cargo/bin:$PATH\"' >> ~/.zshrc` |\n| Install jq on macOS | `brew install jq` |\n| Grep TODO in /var/www | `grep -r \"TODO\" /var/www` |\n\n## Usage\n\n```bash\n# llama.cpp with thinking\nllama-cli -m gemma4-31b-IQ3_XS.gguf -cnv -ngl 99 --ctx-size 8192 \\\n  --reasoning on --reasoning-budget 512\n\n# Ollama\ncat > Modelfile << 'EOF'\nFROM ./gemma4-31b-IQ3_XS.gguf\nPARAMETER temperature 0.1\nPARAMETER num_ctx 8192\nEOF\nollama create gemma4-31b-iq3xs -f Modelfile\n```\n\n## Quantization Details\n\n- **Tool**: llama.cpp (build 0d049d6)\n- **Imatrix**: Computed from 200 chunks of synthetic CLI/package management data\n- **Source**: Converted from google/gemma-4-31b-it safetensors via `convert_hf_to_gguf.py`\n- **Architecture**: Gemma4 with sliding + full attention pattern (every 6th layer is full attention)\n\n## Hardware Requirements\n\n- **Minimum RAM**: 16GB (with partial offload)\n- **Recommended**: Apple Silicon with 32GB+ unified memory\n- **Performance**: ~21 tok/s on M4 Max 128GB with full GPU offload\n\n## Benchmark Methodology\n\n- **NL2Bash test set**: 50 examples from the official deterministic split (RANDOM_SEED=100, fold 11 from `TellinaTool/nl2bash`)\n- **Inference**: Each question runs as an isolated llama-cli subprocess with `-p` prompt, `--reasoning off`, `--ctx-size 4096`, `--max-tokens 200`, `--temp 0.1`\n- **Scoring**: NLTK BLEU corpus-level + character-level F1 + exact match\n- **Reproducibility**: All scripts and JSONL predictions are public at the dataset repo above\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "3-bit",
    "IQ3_XS",
    "cli",
    "conversational",
    "gemma4",
    "imatrix",
    "llama-cpp",
    "package-management",
    "quantized",
    "text-generation",
    "en",
    "license:apache-2.0",
    "endpoints_compatible",
    "region:us"
  ],
  "likes": 0,
  "downloads": 3122,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-15T13:08:44.000Z",
  "created_at": "2026-04-10T18:35:24.000Z",
  "pipeline_tag": "text-generation",
  "library_name": "gguf"
}

{
  "_id": "69d942ecffe75ba004b72bfb",
  "id": "KikoCis/gemma-4-31b-it-IQ3_XS-GGUF",
  "modelId": "KikoCis/gemma-4-31b-it-IQ3_XS-GGUF",
  "sha": "61b90da7bd948635f4e8422752aeb63b04953f4e",
  "createdAt": "2026-04-10T18:35:24.000Z",
  "lastModified": "2026-04-15T13:08:44.000Z",
  "author": "KikoCis",
  "downloads": 3122,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "gguf",
  "siblings_count": 3
}