---

license: mit

---

gguf-server

Launch your own mini LLM server to connect gguf, i.e., for vibe coding task (see below)

!screenshot

install via pip/pip3

pip install gguf-server

Building the bundled engine requires a C/C++ toolchain and CMake ≥ 3.15 (on

Windows: MSVC Build Tools). The engine source is resolved from the vendored

vendor/server copy (see scripts/vendor_engine.py), a sibling ../server

checkout, or GGUF_SERVER_ENGINE_DIR. That tree is self-contained — it carries

its own ggml kernels, GGUF runtime, common layer and HTTP server, with no

llama.cpp checkout and no external ggml — so the build compiles the server

binary and nothing else.

GPU and accelerator backends

The default build is CPU-only. Backends are opt-in and can be requested with

an environment variable or a CMake define — the env var is usually easier to

pass through pip:

GGUF_SERVER_CUDA=1   pip install gguf-server    # NVIDIA (needs the CUDA toolkit)
GGUF_SERVER_HIP=1    pip install gguf-server    # AMD (needs ROCm/HIP)
GGUF_SERVER_VULKAN=1 pip install gguf-server    # cross-vendor (needs the Vulkan SDK)

CMAKE_ARGS="-DGGUF_SERVER_CUDA=ON" pip install gguf-server   # equivalent

Available: CUDA, HIP, VULKAN, METAL. These are the kernels the engine

tree actually ships (vendor/server/kernels/src); the backends llama.cpp has

and this engine does not — SYCL, CANN, OpenCL, WebGPU, RPC, zDNN, ZenDNN,

OpenVINO, Hexagon, MUSA, BLAS — are simply absent. Each option maps to the

ggml option of the same name, and the finer-grained GGML_* knobs

(GGML_NATIVE, GGML_AVX512, GGML_CUDA_FA_ALL_QUANTS, …) can still be

passed straight through as -DGGML_<NAME>=…. **On macOS, Metal is on by

default** (with the shader library embedded in the binary) — no flag needed.

usage

enter GUI panel

gguf-server

connect the model file straight from CLI

gguf-server engine -- --model model.gguf

or run it with gguf-connector

ggc sv

!screenshot