majentik/nemotron-3-nano-30b-a3b-rotorquant-gguf-q5_k_m overview

{
  "metadata": {},
  "card_data": {
    "license": "other",
    "license_name": "nvidia-open-model-license",
    "license_link": "https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf",
    "base_model": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
    "tags": [
      "gguf",
      "rotorquant",
      "kv-cache-quantization",
      "nemotron",
      "nvidia",
      "mamba2",
      "hybrid",
      "moe",
      "llama-cpp",
      "quantized"
    ],
    "library_name": "gguf",
    "pipeline_tag": "text-generation",
    "language": [
      "en"
    ],
    "frontmatter": {
      "license": "other",
      "license_name": "nvidia-open-model-license",
      "license_link": "https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf",
      "base_model": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
      "tags": [
        "gguf",
        "rotorquant",
        "kv-cache-quantization",
        "nemotron",
        "nvidia",
        "mamba2",
        "hybrid",
        "moe",
        "llama-cpp",
        "quantized"
      ],
      "library_name": "gguf",
      "pipeline_tag": "text-generation",
      "language": [
        "en"
      ]
    },
    "hero_image_url": "",
    "summary": "GGUF Q5_K_M weight-quantized variant of nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 optimised for use with **RotorQuant** KV cache compression via a dedicated llama.cpp fork. > **Important:** RotorQuant KV cache types (planar3, iso3) are **not** available in upstream llama.cpp, standard Ollama, or LM Studio. > They require a specific llama.cpp fork. > The GGUF file itself is a standard GGUF and works with any llama.cpp-compatible runtime using normal KV cache types (f16, q8_0, q4_0, etc.).",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: other\nlicense_name: nvidia-open-model-license\nlicense_link: https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf\nbase_model: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16\ntags:\n- gguf\n- rotorquant\n- kv-cache-quantization\n- nemotron\n- nvidia\n- mamba2\n- hybrid\n- moe\n- llama-cpp\n- quantized\nlibrary_name: gguf\npipeline_tag: text-generation\nlanguage:\n- en\n---\n\n# Nemotron-3-Nano-30B-A3B-RotorQuant-GGUF-Q5_K_M\n\nGGUF Q5_K_M weight-quantized variant of [nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) optimised for use with **RotorQuant** KV cache compression via a dedicated llama.cpp fork.\n\n> **Important:** RotorQuant KV cache types (`planar3`, `iso3`) are **not** available in upstream llama.cpp, standard Ollama, or LM Studio.\n> They require a [specific llama.cpp fork](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache).\n> The GGUF file itself is a standard GGUF and works with any llama.cpp-compatible runtime using normal KV cache types (f16, q8_0, q4_0, etc.).\n\n## Overview\n\nThis model combines two independent compression techniques:\n\n| Technique | What it does | Requirement |\n|-----------|-------------|-------------|\n| **GGUF Q5_K_M weight quantization** | Reduces model size from ~60 GB (BF16) to ~20.4 GB | Any llama.cpp-compatible runtime |\n| **RotorQuant KV cache compression** — block-diagonal Clifford-algebra rotors for 3-bit KV cache (`--cache-type-k iso3 --cache-type-v iso3`) | Block-diagonal rotations / random rotation for compressed KV cache | [llama-cpp-turboquant fork](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache) only |\n\n## Quickstart\n\n### Option A — With RotorQuant KV cache (fork required)\n\nYou must build from the RotorQuant-enabled llama.cpp fork:\n\n```bash\n# Clone and build the fork\ngit clone https://github.com/johndpope/llama-cpp-turboquant.git\ncd llama-cpp-turboquant && git checkout feature/planarquant-kv-cache\n\n# CUDA (Windows/Linux)\ncmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j\n\n# Metal (Apple Silicon)\ncmake -B build -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j\n\n# Run with RotorQuant KV cache\n./build/bin/llama-cli -m Nemotron-3-Nano-30B-A3B-RotorQuant-GGUF-Q5_K_M.gguf \\\n  --cache-type-k iso3 --cache-type-v iso3 \\\n  -ngl 99 -fa \\\n  -p \"Explain quantum computing\"\n\n# Or run as a server\n./build/bin/llama-server -m Nemotron-3-Nano-30B-A3B-RotorQuant-GGUF-Q5_K_M.gguf \\\n  --cache-type-k iso3 --cache-type-v iso3 \\\n  -ngl 99 -fa --jinja\n```\n\n### Option B — With standard llama.cpp / LM Studio / Ollama\n\nThe GGUF works as a normal quantised model. You won't get RotorQuant-specific KV cache benefits, but standard KV cache quantization (q8_0, q4_0) still reduces VRAM significantly.\n\n**llama.cpp (upstream)**\n```bash\nllama-cli -m Nemotron-3-Nano-30B-A3B-RotorQuant-GGUF-Q5_K_M.gguf \\\n  --cache-type-k q8_0 --cache-type-v q8_0 \\\n  -ngl 99 -fa \\\n  -p \"Explain quantum computing\"\n```\n\n**LM Studio**\n1. Download the GGUF file and load in LM Studio.\n2. Enable **Developer Mode** (Settings → Developer).\n3. In the model loader's advanced settings, set **Flash Attention** to ON.\n4. Set **K Cache Quantization** and **V Cache Quantization** to `q8_0` (or `q4_0` for more aggressive VRAM savings).\n5. Note: LM Studio does not currently support RotorQuant's `iso3` cache types. Track [this feature request](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1719) for updates.\n\n**Ollama**\n```bash\n# Standard Ollama does not support RotorQuant cache types.\n# Use with default or q8_0 KV cache via OLLAMA_KV_CACHE_TYPE=q8_0\nOLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_FLASH_ATTENTION=1 ollama run majentik/Nemotron-3-Nano-30B-A3B-RotorQuant-GGUF-Q5_K_M\n```\n\n## Specifications\n\n| Property | Value |\n|----------|-------|\n| Base Model | [nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) |\n| Architecture | Mamba-2 + Transformer hybrid Sparse MoE |\n| Parameters | 30.7B total, 3.2B active per token |\n| Context Length | 1M |\n| Weight Quantization | GGUF Q5_K_M (high quality, balanced 5-bit) |\n| Original Size (BF16) | ~60 GB |\n| Quantized File Size | ~20.4 GB |\n| KV Cache (RotorQuant) | 3-bit via `--cache-type-k iso3 --cache-type-v iso3` (fork only) |\n| KV Cache (standard) | q8_0, q4_0, f16, etc. (any llama.cpp runtime) |\n| License | other |\n| Modalities | Text only |\n| Compatible Runtimes | llama.cpp, LM Studio, Ollama, koboldcpp |\n\n## What is RotorQuant?\n\n[RotorQuant](https://github.com/scrya-com/rotorquant) is a KV cache compression method based on Clifford algebra (Cl(3,0)) rotors. It was developed as a faster, more parameter-efficient alternative to Google's [TurboQuant](https://arxiv.org/abs/2504.19874) (ICLR 2026).\n\nInstead of applying a dense d×d random orthogonal rotation matrix (as TurboQuant does), RotorQuant uses lightweight block-diagonal rotations — independent 2D/4D rotations per pair/quartet — achieving O(d) complexity instead of O(d log d), fully parallelisable with no inter-element dependencies.\n\n**Benchmarks from the RotorQuant repository** (Llama 3.1 8B, RTX 5090 — results will vary by model and hardware):\n\n| Metric | RotorQuant (iso3) | TurboQuant | Standard q4_0 |\n|--------|-------------------|------------|---------------|\n| Prefill Speed | 3,822 tok/s | 722 tok/s | — |\n| Decode Speed | 119 tok/s | 93 tok/s | — |\n| Perplexity (PPL) | 6.91 | 7.07 | — |\n| KV Compression | ~5× vs FP16 | ~5× vs FP16 | ~4× vs FP16 |\n| Rotation Parameters | 4 per rotor | 16,384 per matrix | N/A |\n\n> **Note:** These benchmarks are from the RotorQuant repository using Llama 3.1 8B on an RTX 5090. Performance on Nemotron-3-Nano-30B-A3B will differ. Independent benchmarks for this specific model are welcome — please open a discussion if you have results to share.\n\n## Current Status of RotorQuant in the Ecosystem\n\n| Runtime | RotorQuant Support | Standard KV Quant |\n|---------|---------------------|-------------------|\n| llama.cpp (upstream) | ❌ Not merged | ✅ q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 |\n| llama-cpp-turboquant fork | ✅ planar3, iso3 | ✅ All standard types |\n| LM Studio | ❌ [Requested](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1719) | ✅ Via advanced settings |\n| Ollama | ❌ Not supported | ✅ Via OLLAMA_KV_CACHE_TYPE |\n| koboldcpp | ❌ Not supported | ✅ Standard types |\n\n## Recommended Settings\n\nFor VRAM-constrained setups, standard q8_0 KV cache quantization already halves KV cache memory with negligible quality impact. Flash Attention should always be enabled — it is required for V cache quantization and improves memory efficiency regardless.\n\n| VRAM | Suggested Configuration |\n|------|------------------------|\n| 24 GB (RTX 4090) | Q5_K_M + q8_0 KV cache + Flash Attention, 8K–16K context |\n| 16 GB | Q5_K_M + q4_0 KV cache + Flash Attention, 4K–8K context |\n| 48+ GB | Q5_K_M + f16 KV cache, full 32K+ context |\n\n## See Also\n\n- [RotorQuant GitHub](https://github.com/scrya-com/rotorquant)\n- [llama-cpp-turboquant fork](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache)\n- [TurboQuant llama.cpp discussion](https://github.com/ggml-org/llama.cpp/discussions/20969)\n- [TurboQuant paper (arXiv: 2504.19874)](https://arxiv.org/abs/2504.19874)\n- [Base model: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16)\n- [Nemotron-3-Nano-30B-A3B announcement](https://huggingface.co/blog/nvidia/nemotron-3-nano-efficient-open-intelligent-models)\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "rotorquant",
    "kv-cache-quantization",
    "nemotron",
    "nvidia",
    "mamba2",
    "hybrid",
    "moe",
    "llama-cpp",
    "quantized",
    "text-generation",
    "en",
    "arxiv:2504.19874",
    "base_model:nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
    "base_model:quantized:nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
    "license:other",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 97,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-17T01:17:12.000Z",
  "created_at": "2026-04-13T21:40:47.000Z",
  "pipeline_tag": "text-generation",
  "library_name": "gguf"
}

{
  "_id": "69dd62df7732fbc62b7112e2",
  "id": "majentik/Nemotron-3-Nano-30B-A3B-RotorQuant-GGUF-Q5_K_M",
  "modelId": "majentik/Nemotron-3-Nano-30B-A3B-RotorQuant-GGUF-Q5_K_M",
  "sha": "c595e7ebef43489eeba137d041a8b66c6225f781",
  "createdAt": "2026-04-13T21:40:47.000Z",
  "lastModified": "2026-04-17T01:17:12.000Z",
  "author": "majentik",
  "downloads": 97,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "gguf",
  "siblings_count": 3
}