---

license: mit

tags:

- kernel

- gguf

- quantization

---

GGUF kernels

Compute directly on the packed blocks of a GGUF checkpoint, so a quantized model runs without ever

materializing a dense copy of its weights.

Ops

| op | signature |

| --- | --- |

| dequantize | (blocks, ggml_type, rows, cols, dtype) -> (rows, cols) |

| mul_mat_vec | (blocks, x, ggml_type, out_features) -> (rows, out_features) f32, fused dequant-gemv |

blocks is a GGUF weight exactly as stored: (out_features, bytes_per_row) uint8. mul_mat_vec

takes at most MAX_GEMV_ROWS rows of x; above that, unpack with dequantize and use a dense

matmul. The quant types each backend implements a gemv for are in GEMV_TYPES.

Devices

| backend | torch | targets |

| --- | --- | --- |

| CUDA | 2.11, 2.12 | x86_64-linux, cu126/128/130/132, sm 7.5–12.0 |

| Metal | 2.12, 2.13 | aarch64-darwin |

Where the kernels come from

llama.cpp's ggml, vendored rather than reimplemented.

vendor/UPSTREAM records the revision.

CUDA compiles ggml-cuda's sources; Metal compiles ggml-metal.metal into the embedded metallib.

Only the files listed in build.toml's src are built — the rest of the tree rides along so a pin

bump cannot leave a dangling include.

Updating

python vendor.py --rev <llama.cpp commit>   # re-vendor, updates vendor/UPSTREAM
nix run .#build-and-copy                    # rebuild every variant into build/

vendor.py copies whole trees plus one rename: upstream's mmvq.cu lands as mmvq-impl.cuh,

because gguf_cuda/ggml_dispatch.cu #includes it instead of compiling it as its own translation

unit.