kikocis/gemma-4-31b-otter-v3-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "tags": [
      "3-bit",
      "IQ3_XS",
      "Q3_K_M",
      "ablation-study",
      "cli",
      "conversational",
      "gemma4",
      "gguf",
      "imatrix",
      "layer-pruned",
      "llama-cpp",
      "package-management",
      "quantized",
      "text-generation"
    ],
    "base_model": "google/gemma-4-31b-it",
    "model_type": "gemma4",
    "quantized_by": "KikoCis",
    "pipeline_tag": "text-generation",
    "language": [
      "en"
    ],
    "base_model_relation": "quantized",
    "library_name": "gguf",
    "frontmatter": {
      "license": "apache-2.0",
      "tags": [
        "3-bit",
        "IQ3_XS",
        "Q3_K_M",
        "ablation-study",
        "cli",
        "conversational",
        "gemma4",
        "gguf",
        "imatrix",
        "layer-pruned",
        "llama-cpp",
        "package-management",
        "quantized",
        "text-generation"
      ],
      "base_model": "google/gemma-4-31b-it",
      "model_type": "gemma4",
      "quantized_by": "KikoCis",
      "pipeline_tag": "text-generation",
      "language": [
        "en"
      ],
      "base_model_relation": "quantized",
      "library_name": "gguf"
    },
    "hero_image_url": "",
    "summary": "> ## 🏆 Smaller & More Capable: gemma-4-31b-it-IQ2_M-GGUF > > If you want the smallest Gemma 4 31B variant that preserves (or beats) f16 quality, use our **custom IQ2_M at 10.17 GB** — it scores **F1 84.71%** and **BLEU-4 22.39** on NL2Bash (**beats f16 on BLEU-4**, 6× smaller). > Otter v3 below is kept as the research artifact for layer-pruning without recovery fine-tuning. The IQ2_M repo is the practical deployment choice. --- **Otter v3** is Google's Gemma 4 31B IT with **2 surgically removed transformer layers** (layers 27 and 29), identified through a 3-phase empirical study. The remaining 58 layers are quantized to GGUF without any fine-tuning recovery.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\ntags:\n- 3-bit\n- IQ3_XS\n- Q3_K_M\n- ablation-study\n- cli\n- conversational\n- gemma4\n- gguf\n- imatrix\n- layer-pruned\n- llama-cpp\n- package-management\n- quantized\n- text-generation\nbase_model: google/gemma-4-31b-it\nmodel_type: gemma4\nquantized_by: KikoCis\npipeline_tag: text-generation\nlanguage:\n- en\nbase_model_relation: quantized\nlibrary_name: gguf\n---\n\n# Otter v3 — Gemma 4 31B Surgically Pruned (58 layers)\n\n> ## 🏆 Smaller & More Capable: [gemma-4-31b-it-IQ2_M-GGUF](https://huggingface.co/KikoCis/gemma-4-31b-it-IQ2_M-GGUF)\n>\n> If you want the smallest Gemma 4 31B variant that preserves (or beats) f16 quality, use our **custom IQ2_M at 10.17 GB** — it scores **F1 84.71%** and **BLEU-4 22.39** on NL2Bash (**beats f16 on BLEU-4**, 6× smaller).\n> Otter v3 below is kept as the research artifact for layer-pruning without recovery fine-tuning. The IQ2_M repo is the practical deployment choice.\n\n---\n\n**Otter v3** is Google's Gemma 4 31B IT with **2 surgically removed transformer layers** (layers 27 and 29), identified through a 3-phase empirical study. The remaining 58 layers are quantized to GGUF without any fine-tuning recovery.\n\n## Why This Model (research artifact)\n\nA 3-phase layer study found:\n- **Layer 27** (sliding attention): single-layer ablation REDUCES perplexity by **74%**\n- **Layer 29** (full attention): single-layer ablation REDUCES perplexity by **71%**\n- **Block ablation {27, 29}**: PPL drops from 496 → **117** (-76%)\n\nThese layers were adding noise. Otter v3 is a research demonstration that some layers can be surgically removed with positive effect on perplexity — but dynamic quantization (see IQ2_M above) dominates it on downstream benchmarks.\n\n## Files\n\n| File | Size | BPW | CLI 7/7 (thinking) | NL2Bash F1 (no-think) |\n|------|------|-----|--------|------|\n| `otter-v3-Q3_K_M.gguf` | 14.1 GB | 3.98 | **7/7** ✓ | 79.89% |\n| `otter-v3-IQ3_XS.gguf` | 12.1 GB | 3.41 | 6/7 | 78.72% |\n\n## NL2Bash Benchmark Results — Full Comparison\n\n50 examples from the official NL2Bash test split (Stanford/Tellina), reasoning OFF:\n\n| Model | Size | Layers | Char-F1 | BLEU-1 | BLEU-4 | EM |\n|-------|------|--------|---------|--------|--------|-----|\n| Unsloth UD-IQ3_XXS | 11.84 GB | 60 | **85.06%** | **46.30** | 20.26 | 12% |\n| Base f16 (full precision) | 61.4 GB | 60 | 84.76% | 43.94 | 21.02 | 12% |\n| Base IQ3_XS (sibling repo) | 13.1 GB | 60 | 84.76% | 42.77 | 18.95 | 8% |\n| **✨ IQ2_M (CLI imatrix, recommended)** | **10.17 GB** | 60 | **84.71%** | **44.72** | **22.39** | **12%** |\n| Unsloth UD-IQ2_M | 10.75 GB | 60 | 84.02% | 42.38 | 18.64 | 10% |\n| Otter-v3 Q3_K_M (this repo) | 14.1 GB | 58 | 79.89% | 35.09 | 12.80 | 0% |\n| Otter-v3 IQ3_XS (this repo) | 12.1 GB | 58 | 78.72% | 37.01 | 15.14 | 4% |\n| Base Q2_K (3rd party) | 11.0 GB | 60 | 58.60% | 18.82 | 6.92 | 0% |\n\n**Honest conclusion**: On NL2Bash, **layer pruning without recovery training is dominated by dynamic quantization**. Our custom IQ2_M at 10.17 GB scores **6 F1 points higher** than Otter-v3 IQ3_XS at 12.1 GB. Keep Otter as a study exhibit; deploy the IQ2_M.\n\n**Full study + raw benchmark data**: https://huggingface.co/datasets/KikoCis/gemma4-31b-layer-study\n**Interactive site**: https://kikocisbot.github.io/gemma4-31b-study (and soon https://otter.utopiaia.com)\n\n## CLI Benchmark (7/7 with thinking)\n\n| Test | Otter v3 Q3_K_M |\n|------|------------|\n| Install neofetch on Void Linux | `sudo xbps-install -S neofetch` ✓ |\n| Install htop on Ubuntu | `sudo apt update && sudo apt install htop` ✓ |\n| Search ripgrep on Arch | `pacman -Ss ripgrep` ✓ |\n| Search packages on Void | `xbps-query -Rs <package>` ✓ |\n| Add cargo to PATH in zsh | (multi-step zshrc edit) ✓ |\n| Install jq on macOS | `brew install jq` ✓ |\n| Grep TODO in /var/www | `grep -r \"TODO\" /var/www` ✓ |\n\n## Layer Analysis Methodology\n\n### Phase A — Block Influence (BI Score)\nMeasured `1 - cosine_similarity(h_in, h_out)` for each layer across 300 prompts in 6 categories.\n\n### Phase B — Logit Lens\nProjected each layer's hidden state through the final norm + lm_head. Decision happens in layers 56-59.\n\n### Phase C — Single & Block Ablation\nDisabled each layer individually and measured perplexity on held-out text. Identified layers 27 and 29 as net-negative.\n\n**Full study + raw measurements + interactive visualization**: https://huggingface.co/datasets/KikoCis/gemma4-31b-layer-study\n\n## Usage\n\n```bash\n# llama.cpp with thinking (recommended for best results with this model)\nllama-cli -m otter-v3-Q3_K_M.gguf -cnv -ngl 99 --ctx-size 8192 \\\n  --reasoning on --reasoning-budget 512\n```\n\n## Quantization Details\n\n- **Tool**: llama.cpp (build 0d049d6)\n- **Q3_K_M**: No imatrix (mixed q3_K/q4_K/q5_K)\n- **IQ3_XS**: Custom imatrix computed from 200 chunks on the 58-layer model\n- **Source**: Custom 58-layer safetensors (base model minus layers 27, 29)\n\n## Hardware\n\n- **Recommended RAM**: 16GB+ (Q3_K_M) or 14GB+ (IQ3_XS)\n- **Apple Silicon**: ~22 tok/s on M4 Max with full GPU offload\n\n## Limitations\n\n- **Requires reasoning enabled** for best results on complex tasks\n- Without thinking, loses ~5 F1 vs base on NL2Bash (dominated by IQ2_M)\n- 58-layer architecture is non-standard; some frameworks may need updates\n- **No fine-tuning recovery applied** — pure structural pruning\n\n## Related Resources\n\n- **[Gemma 4 31B IT — IQ2_M (CLI imatrix, 10.17 GB, F1 84.71%)](https://huggingface.co/KikoCis/gemma-4-31b-it-IQ2_M-GGUF)** ⭐ recommended\n- [Base IQ3_XS (60-layer, 13.1 GB, F1 84.76%)](https://huggingface.co/KikoCis/gemma-4-31b-it-IQ3_XS-GGUF)\n- [Full layer analysis study](https://huggingface.co/datasets/KikoCis/gemma4-31b-layer-study) — raw data + scripts + interactive visualization\n- [Interactive study site](https://kikocisbot.github.io/gemma4-31b-study)\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "3-bit",
    "IQ3_XS",
    "Q3_K_M",
    "ablation-study",
    "cli",
    "conversational",
    "gemma4",
    "imatrix",
    "layer-pruned",
    "llama-cpp",
    "package-management",
    "quantized",
    "text-generation",
    "en",
    "license:apache-2.0",
    "endpoints_compatible",
    "region:us"
  ],
  "likes": 1,
  "downloads": 2037,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-15T13:19:49.000Z",
  "created_at": "2026-04-12T09:11:46.000Z",
  "pipeline_tag": "text-generation",
  "library_name": "gguf"
}

{
  "_id": "69db61d2335901ef3e15a02f",
  "id": "KikoCis/gemma-4-31b-otter-v3-GGUF",
  "modelId": "KikoCis/gemma-4-31b-otter-v3-GGUF",
  "sha": "019556d07dea096c78b032c3d291ce2b1dee1e2d",
  "createdAt": "2026-04-12T09:11:46.000Z",
  "lastModified": "2026-04-15T13:19:49.000Z",
  "author": "KikoCis",
  "downloads": 2037,
  "likes": 1,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "gguf",
  "siblings_count": 4
}