---

license: apache-2.0

base_model: UltimateIntent/Gemory-26B-A4B-GGUF

tags:

• text-generation
• gemma4
• gguf
• nvfp4
• fp4
• blackwell
• rtx-50-series
• roleplay
• creative-writing
• erp
• ablitterated
• sillytavern
• llama-cpp
• koboldcpp

language:

• en

pipeline_tag: text-generation

library_name: gguf

---

Gemory-26B-A4B — Blackwell NVFP4 GGUF

This is the first native GGUF release of UltimateIntent/Gemory-26B-A4B in NVIDIA Blackwell NVFP4 format (GGML_TYPE_NVFP4).

Designed specifically for local deployment in llama.cpp and kobold.cpp, this quantization achieves high mathematical fidelity and fine-grained micro-scaling (group size 16) while fitting strictly within 16 GB VRAM consumer GPUs (e.g., RTX 5060 Ti / RTX 5070 / RTX 5080).

---

Model Details

• Base Model: UltimateIntent/Gemory-26B-A4B
• Architecture: Gemma 4 MoE (26B total parameters, ~4B active per token)
• Quantization: Native Blackwell NVFP4 (E2M1 weights, UE4M3 scales per 16-element sub-blocks, repacked into 64-element GGML super-blocks)
• Router Precision: Full BF16 router projection matrices preserved for 100% expert gating coherence
• Quant File Size: 15.3 GB

---

Benchmark Highlights (Tested on RTX 5060 Ti 16GB, Vulkan Backend)

• Prompt Processing (pp128): 1,017.68 ± 91.58 tokens/sec via Vulkan Cooperative Matrix 2 (NV_coopmat2).
• Token Generation (tg32): 65.11 ± 0.15 tokens/sec.
• Total VRAM Footprint: 14.24 GiB (leaves headroom for OS/desktop compositor without crashing).

---

Why NVFP4 over Standard Q4_0?

1. Logarithmic Weight Representation (E2M1): Unlike linear integer quantization (Q4_0), FP4 non-linear spacing matches the normal Gaussian bell-curve of neural network weights, placing high density around zero.
2. Micro-Scaling (Group Size 16): 1 scale factor per 16 weights (compared to 32 in standard Q4_0), preserving emotional nuance, creative vocabulary, and character card adherence in SillyTavern without repetitive cliché loops.

---

How to Run

1. With llama.cpp (b10819+)

Ensure you have a recent build of llama.cpp with Vulkan or CUDA support:

llama-cli \
  -m Gemory-26B-A4B-nvfp4.gguf \
  -ngl 99 \
  -c 4096 \
  -p "<|turn>user\nHello! Tell me a creative story.\n<turn|>\n<|turn>model\n"

2. With llama-server (For SillyTavern / Open WebUI)

llama-server \
  -m Gemory-26B-A4B-nvfp4.gguf \
  -ngl 99 \
  -c 8192 \
  --host 0.0.0.0 --port 8080

In SillyTavern, set API to Text Completion (LLaMA.cpp) or Chat Completion (OpenAI compatible) pointing to http://localhost:8080.

---

Acknowledgments & Credits

• Base model training & fine-tuning: UltimateIntent
• Quantized and converted with custom patches by Shadowplague