---

tags:

- gguf

- conversational

- mxfp4

- vision

library_name: llama.cpp

base_model: google/gemma-4-E4B-it

license: apache-2.0

---

Gemma 4 E4B IT — MXFP4 GGUF

> Quantized GGUF release of google/gemma-4-E4B-it by FreedomAISVR. Ready for local inference with llama.cpp.

About the Base Model

| Property | Value |

|---|---|

| Model | google/gemma-4-E4B-it |

| Architecture | Gemma4ForConditionalGeneration (Vision + Text) |

| Parameters | ~3B (E4B = Efficient 4B-class) |

| Context Length | 131,072 tokens (128K) |

| Vision Support | Yes — image and video input via mmproj |

| License | Apache 2.0 |

| Developer | Google DeepMind |

Quantization Details

| Property | Value |

|---|---|

| Format | MXFP4 (OCP Microscaled FP4) |

| Bits Per Weight | 5.37 BPW |

| GGUF File Size | 4.8 GB |

| F16 Source | 15.0 GB (720 tensors) |

Files

| File | Size | Description |

|---|---|---|

| gemma-4-e4b-it-mxfp4.gguf | 4.8 GB | MXFP4 quantized text model |

| mmproj-gemma-4-E4B-it-f16.gguf | 945 MB | F16 multimodal projector (vision encoder) |

Performance (RTX 5060 Ti 16GB, CUDA 13.2)

| Metric | Value |

|---|---|

| Generation Speed | 97.1 tokens/sec |

| Prompt Processing | 175.1 tokens/sec |

| Context Length | 128K (Q8_0 KV cache) |

| VRAM Usage | ~7 GB total (model + KV + vision) |

Recommended llama.cpp Setup

These are the exact flags used to achieve the benchmarked speeds. Tested on RTX 5060 Ti 16GB with CUDA 13.2 and llama.cpp latest master.

Build llama.cpp (Windows)

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build --config Release -j

Run with Full GPU Offload + Flash Attention + 128K Context

llama-cli -m gemma-4-e4b-it-mxfp4.gguf \
    -ngl 99 \
    --flash-attn on \
    -c 131072 \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --parallel 1 \
    --jinja

Flag Explanation

| Flag | Purpose |

|---|---|

| -ngl 99 | Offload all layers to GPU |

| --flash-attn on | Enable Flash Attention (required for long context) |

| -c 131072 | Set context to 128K tokens |

| --cache-type-k q8_0 | Quantize KV key cache to Q8_0 (saves VRAM, near-lossless) |

| --cache-type-v q8_0 | Quantize KV value cache to Q8_0 |

| --parallel 1 | Single sequence (saves VRAM) |

| --jinja | Enable Jinja2 chat template rendering |

Using the Vision Model (mmproj)

llama-server -m gemma-4-e4b-it-mxfp4.gguf \
    --mmproj mmproj-gemma-4-E4B-it-f16.gguf \
    -ngl 99 \
    --flash-attn on \
    -c 131072 \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --port 8080

Then visit http://localhost:8080 for the web UI with image upload support.

Using with llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="gemma-4-e4b-it-mxfp4.gguf",
    n_gpu_layers=99,
    flash_attn=True,
    n_ctx=131072,
    verbose=False,
)

output = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello!"}]
)
print(output["choices"][0]["message"]["content"])

Hardware Requirements

• GPU: NVIDIA RTX 50 series (Blackwell) recommended for MXFP4 acceleration
• VRAM: 8GB+ recommended (128K context with Q8_0 KV)
• RAM: 16GB+ system RAM
• Storage: 4.8 GB free space

License

This model is a quantized release of google/gemma-4-E4B-it, which is licensed under Apache 2.0. This quantized version inherits the same license.

Links

• Base Model: google/gemma-4-E4B-it
• FreedomAISVR: HuggingFace
• llama.cpp: GitHub