---

base_model: parmanu-lcs2/Phi-4-11B

library_name: gguf

pipeline_tag: text-generation

tags:

• gguf
• llama.cpp
• phi-4
• pruned
• sniper
• local-llm

---

Phi-4-11B-GGUF

This is a GGUF build of parmanu-lcs2/Phi-4-11B, a SNIPER-pruned Phi-4 model with about 25% target compression. The source contains about 11.02B parameters after pruning, and its recovery LoRA was merged into the model weights by the source author.

This build is for users who want a smaller local Phi-4-family model and are comfortable running a patched llama.cpp build.

Read this first

This is not a drop-in GGUF for stock llama.cpp yet. The source model uses a custom pruned Phi architecture, so current stock llama.cpp and GUI apps that bundle stock llama.cpp may fail to load it. Use the tested patch and commands below.

The published file is intentionally more conservative than a normal Q4_K_M. A smaller standard Q4_K_M candidate loaded successfully, but introduced extra generation regressions that were not present in the source/BF16 behavior. That candidate was rejected. The published Q4_K_M stability profile keeps sensitive tensor groups at higher precision.

Recommended setup

For a 24 GB NVIDIA GPU, use the patched llama.cpp build, full GPU offload, and start at 4,096 context.

For a 12 GB GPU, use partial offload rather than assuming full offload will fit comfortably. Start around -ngl 20 to -ngl 24 and adjust based on VRAM use.

What behavior should you expect?

Normal short chat, system/user separation, multi-turn chat, and the tested arithmetic prompt work normally with the published file.

The source model itself has some quirks on synthetic prompts. Strings such as SMOKE_OK / TEMPLATE_OK can trigger repetition, and the original source model itself answers 37 * 23 incorrectly. Those behaviors were reproduced directly from the original safetensors with the source custom Python runtime, so they are not Q4-specific failures.

If you want a GGUF that loads directly in unmodified llama.cpp or LM Studio, this repository is not yet the right build for that workflow.

What you get

• File: Phi-4-11B-Q4_K_M.gguf
• Profile: Q4_K_M stability profile
• Size: 10.70 GiB / 11,494,478,976 bytes
• Effective density: about 8.34 bpw
• Source context limit: 16,384 tokens
• Recommended starting context: 4,096 tokens
• Chat template: embedded in the GGUF
• Tested backend: patched llama.cpp with CUDA

Memory / VRAM expectations

The GGUF is 10.70 GiB. On the validation RTX 4090 at -c 4096, llama.cpp projected 11,551 MiB CUDA use: 10,682 MiB model + 740 MiB context + 129 MiB compute.

A 24 GB GPU is a comfortable target for full offload. A 12 GB GPU is close to the edge at this context size; partial offload plus system RAM is safer. For CPU or partial-offload use, 16 GiB RAM is the practical minimum, with 24 GiB or more preferred.

How to run

> Important: this model currently requires the patched llama.cpp support described in the Compatibility section below. Stock upstream llama.cpp does not yet understand this pruned Phi layout.

Download

hf download ramgpt/Phi-4-11B-GGUF Phi-4-11B-Q4_K_M.gguf --local-dir .

llama-cli on Linux

For a 24 GB GPU:

./build/bin/llama-cli \
  -m Phi-4-11B-Q4_K_M.gguf \
  -ngl 999 \
  -c 4096 \
  --conversation

For GPU + RAM split offload, lower the number of GPU layers. For example:

./build/bin/llama-cli \
  -m Phi-4-11B-Q4_K_M.gguf \
  -ngl 24 \
  -c 4096 \
  --conversation

For CPU-only inference:

./build/bin/llama-cli \
  -m Phi-4-11B-Q4_K_M.gguf \
  -ngl 0 \
  -c 4096 \
  --conversation

llama-server

./build/bin/llama-server \
  -m Phi-4-11B-Q4_K_M.gguf \
  -ngl 999 \
  -c 4096 \
  --host 127.0.0.1 \
  --port 8080

The embedded chat template is used automatically by /v1/chat/completions; no separate chat-template file is required.

OpenAI-compatible API

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "local",
    "messages": [
      {"role": "system", "content": "Answer concisely."},
      {"role": "user", "content": "What is 17*19?"}
    ],
    "temperature": 0.2
  }'

Open WebUI

Run the patched llama-server first, then add it in Settings → Admin → Connections → OpenAI-compatible.

• Native/local Open WebUI: http://127.0.0.1:8080/v1
• Open WebUI in Docker: usually http://host.docker.internal:8080/v1
• API key: leave blank unless you started llama-server with authentication

If Open WebUI is in Docker and cannot reach the host, expose llama-server on 0.0.0.0 and configure Docker host networking carefully. Do not expose an unauthenticated llama-server to an untrusted network.

LM Studio and other GUIs

The reliable GUI path today is patched llama-server + an OpenAI-compatible frontend such as Open WebUI.

LM Studio normally loads GGUF models through its own bundled runtime and also exposes its own OpenAI-compatible server. Because this model requires pruned-Phi runtime changes that stock upstream llama.cpp does not yet contain, direct loading in LM Studio may fail until LM Studio ships equivalent support. Do not assume that “latest LM Studio” is sufficient for this GGUF yet.

Context recommendation

Start with -c 4096; that is the context used for the published smoke validation. The source metadata allows up to 16,384 tokens. Increase to 8,192 or 16,384 only when you need the longer window and have verified memory headroom and behavior for your workload.

⚠️ Compatibility: patched llama.cpp required

As of 2026-09-26, stock ggml-org/llama.cpp upstream does not include the support needed for this model's PrunedPhi3ForCausalLM layout.

Verified upstream revision checked before publication:

95887577ab5fead779581a7030a83c7752ff3234

This repository includes:

llama.cpp-pruned-phi3.patch

The patch was developed against:

bd4f514db14d87fded667787a7a963bfbaa98e89

and was also verified to apply cleanly to upstream 95887577ab5fead779581a7030a83c7752ff3234.

Build the tested patched runtime

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 95887577ab5fead779581a7030a83c7752ff3234

hf download ramgpt/Phi-4-11B-GGUF llama.cpp-pruned-phi3.patch --local-dir .
git apply llama.cpp-pruned-phi3.patch

cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-cli llama-server llama-quantize -j

If git apply fails on a newer upstream revision, check out the tested revision above instead of forcing the patch. Once equivalent support is merged upstream, this special step will no longer be necessary.

Common errors

Model fails during load with missing tensors or FFN shape errors

You are probably using stock llama.cpp. Apply the included patch and rebuild.

The final layer appears to have no MLP tensors

That is expected for this checkpoint. The last retained runtime block has n_ff = 0; the patched runtime keeps the source model's post-attention RMSNorm + residual path and does not create nonexistent MLP tensors.

Out of memory

Lower context first, for example from -c 8192 to -c 4096. If needed, reduce GPU offload with -ngl 24, -ngl 16, or lower.

Open WebUI cannot connect

If Open WebUI runs in Docker, 127.0.0.1 points at the container, not your host. Use host.docker.internal or an appropriate host/network address.

The patch does not apply

Use the tested upstream revision 95887577ab5fead779581a7030a83c7752ff3234 rather than forcing hunks onto an unknown newer tree.

Technical Notes

Source and provenance

• Source model: parmanu-lcs2/Phi-4-11B
• Source revision: 22883a073fe2886a3817e0925a58f72d378c996e
• Source safetensors SHA256: 85e2fbffc6d066105b949ef8e1db2f0ab2663c5e084f957abb6a9a3e7f3ab9a8
• Source safetensors size: 22,039,598,904 bytes
• Architecture: PrunedPhi3ForCausalLM
• BF16 GGUF size: 22,043,894,912 bytes
• Q4_K_M SHA256: f91d4ad1cb77fc18fa38d21941a572389b1f7d8302819299cabe6e6495434acb

Pruned structure

The source config declares 40 layers, but the checkpoint actually retains 37 runtime blocks. The retained source-layer mapping is original layers 0–35 plus original layer 37.

Per-layer FFN widths written into GGUF:

[17920, 17737, 17163, 15328, 15328, 14394, 13347, 12940, 12822,
 12888, 12907, 13079, 12933, 13042, 13406, 12965, 12849, 12920,
 12849, 12682, 12836, 12773, 12739, 12985, 12709, 12888, 12901,
 12822, 13029, 12965, 13067, 13104, 12933, 13142, 14329, 17920, 0]

The final retained block has no MLP. The runtime therefore uses hparams.n_ff(i) per layer and skips MLP tensor creation when n_ff(i) == 0.

Tokenizer compatibility normalization

The upstream tokenizer metadata names the tokenizer class TokenizersBackend, which is produced by a newer Transformers stack and was not recognized by the conversion environment used here. For conversion only, the local source metadata was normalized to GPT2TokenizerFast.

The tokenizer vocabulary, special tokens, EOS token and embedded chat template were preserved; this normalization is a converter compatibility fix, not a tokenizer redesign.

Quantization detail

This is a Q4_K_M stability profile, not a pure stock Q4_K_M mix. The base profile is Q4_K_M, while ffn_up, attn_qkv, and attn_output are kept at Q6_K. Irregular ffn_down widths use llama-quantize safety fallbacks, mostly F16 with Q8_0/Q5_0 where shape permits. The rejected standard Q4_K_M candidate was 9.08 GiB; the stability profile is 10.70 GiB and is the exact file that passed the accepted smoke suite.

Runtime / conversion changes

The included llama.cpp patch adds:

• PrunedPhi3ForCausalLM conversion routing
• retained block count derived from the actual pruned structure
• per-layer phi3.feed_forward_length GGUF metadata
• runtime use of per-layer hparams.n_ff(i)
• no MLP tensor construction for n_ff(i) == 0
• preservation of the source custom forward behavior around post-attention RMSNorm and residual handling

Source runtime parity

The original safetensors were executed directly with the source PrunedPhi3ForCausalLM code under Transformers 5.17.0. Transformers 4.57.6 is too old for this custom forward and fails at create_causal_mask(inputs_embeds=...).

The upstream model itself repeats synthetic SMOKE_OK / TEMPLATE_OK prompts and answers 37 * 23 as 90. Those behaviors are therefore source-model limitations, not Q4 regressions. Normal exact responses such as OK and Paris are followed upstream by token 100257 <|endoftext|>; patched llama.cpp recognizes both 100257 and 100265 <|im_end|> as EOG.

Smoke validation

The exact published Q4_K_M SHA passed a real patched llama-server run on an RTX 4090 at -c 4096.

Validated through /v1/chat/completions:

• model load and /health
• 37 runtime layers
• per-layer FFN metadata including final n_ff = 0
• exact short response: OK
• factual short response: Paris
• arithmetic: 17 * 19 = 323
• system/user separation: SYSOK:4.
• multi-turn chat separation: codeword retrieval returned ORANGE.
• normal short generation
• all published smoke generations ended with finish_reason = stop
• no raw <|im_start|>, <|im_end|>, role markers, NaN, or obvious garbage leakage

Synthetic SMOKE_OK / TEMPLATE_OK repetition was reproduced directly on the original upstream safetensors under Transformers 5.17.0. It is documented as a source-model instruction-following limitation and is not counted as a successful quality test.

Credits

Model architecture and weights come from parmanu-lcs2/Phi-4-11B. SNIPER pruning and recovery fine-tuning credit belongs to the source authors. This repository provides the GGUF conversion, compatibility patch, validation and usage guidance.