---

license: apache-2.0

base_model: openai/gpt-oss-120b

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- moe

- pruned

- gpt-oss

- llama.cpp

---

gpt-oss-78b — Uniform Expert-Pruned Experiment

**A structural-pruning experiment on openai/gpt-oss-120b: drop the least-used 1/3 of

experts from every MoE layer, uniformly, with no recovery training, and see what it

actually costs.** Not an APEX/precision-reallocation quant — see below for why that

approach doesn't apply to this model.

APEX quantization takes advantage of the fact that 16bit precision is not actually

needed for most tensors in LLMs. Most tensors can function with 8 bits with effectively

zero degradation and medium-sized models in the 100-200B parameter range have started

distributing 4-bit versions of their models. gpt-oss 120B is one such model where the

BF16 was never actually even distributed - even their safetensors models are distributed

with 4-bit experts. However, Laguna and Nemotron for example offer a 4-bit version

directly from the vendor. APEX extends the speed and size advantages of this lower

quantization to models for which they are not available. This is not a real option for

gpt-oss, where the tensors are already 4-bit.

What this is

gpt-oss-120b is 36 transformer layers × 128 experts/layer (top-4 routing). Since

intermediate_size == hidden_size == 2880, essentially the entire model (~115B of

116.8B real params) lives in the expert FFN stacks — attention and everything else is a

rounding error by comparison. NVIDIA's own gpt-oss-puzzle-88B compresses this same base

model via heterogeneous Puzzle NAS: a different expert count, and even a different

expert width, chosen per layer, plus knowledge distillation, RL, and window-attention

swaps to recover quality afterward. This architecture does succeed in reducing the size,

but currently requires a special kernel to run which isn't even available in stock LLM

frameworks.

This experiment asks the simpler question: **what happens if you skip all of that and

just uniformly drop the same fraction of experts from every layer**, choosing which

specific experts to drop per-layer from real measured usage (not a fixed heuristic), and

apply zero recovery training?

Method

1. Measure real per-expert usage. Hooked each layer's expert-dispatch call during a

real forward pass over a diverse calibration set (web prose + permissive code +

multilingual text — not just English wiki prose, since MoE experts often specialize

by domain, and a narrow calibration corpus would bias "usage" against experts a

different domain actually needs).

1. Drop the least-used 43 of 128 experts per layer (uniform count, not uniform

which experts — each layer's own ranking decides which 85 it keeps), remap the

router accordingly.

1. Pure tensor surgery, no dequantization. The model ships natively in MXFP4

(block-quantized, ~4.25 bits/weight). Every expert-indexed tensor (gate_up_proj/

down_proj blocks, scales, biases, and the router itself) has the expert dimension as

its leading axis, so pruning is a plain index_select on packed MXFP4 bytes — the

quantized values themselves are never touched or decoded.

1. No recovery training. Verified this isn't skippable for free: the native

MXFP4 inference kernels (Triton matmul_ogs) have no registered backward pass at all

— gradient can't reach any router, let alone the pruned experts, without dequantizing

at least part of the model to bf16. Full dequant needs ~156GB just for the pruned

model's weights alone, well past what fits on the hardware this was built on. A

partial fix (dequantize only a small tail of layers to trainable bf16, keep the rest

frozen in fast native format) is architecturally sound but wasn't built out — this

release is the raw, undistilled structural cut.

Results (real, not simulated)

All numbers below are real llama-perplexity/llama-server runs, validated on **two

independent llama.cpp implementations** (mainline ggml-org/llama.cpp and a

Puzzle-support fork) agreeing to within ~1%, same corpus, same settings throughout.

| | original 120b | pruned 78B | native 20b (for reference) |

|---|---:|---:|---:|

| total params | 116.8B | 78.3B | 20.9B |

| layers × experts/layer | 36 × 128 | 36 × 85 | 24 × 32 |

| PPL, raw wikitext completion (ctx 512) | 213.65 | 273.36 | 159.93 |

| chained tool-calling gate (3 trials) | 3/3 | 3/3 | 3/3 |

| hard-tier coding benchmark (5 real tasks, partial credit) | not tested | 47/59 (79.7%), 4/5 fully solved | not tested |

Raw completion PPL degrades ~28% relative to the original with no recovery applied —

a real, non-trivial cost, exactly what you'd expect from skipping the

distillation/RL step that NVIDIA's actual Puzzle pipeline uses. But two things are worth

noting:

• Tool-calling and coding are untouched. The core agentic behavior gpt-oss is

actually built for — chained tool calls with correct argument passing, not falling for

a distractor tool, and solving real multi-file coding challenges — survives the

pruning intact. 4 of 5 hard-tier coding challenges were solved completely, and the

fifth got real partial credit, not a collapse.

• Raw wikitext PPL is a poor proxy for this model family. The native 20b variant

scores better on this metric than either 120b configuration, despite being ~4x

smaller — gpt-oss is heavily RL/chat-tuned, not built for bare next-token completion on

generic prose, so this metric mostly measures something other than what these models

are actually evaluated on in practice.

• Honest bottom line: if the goal is "smallest usable gpt-oss for constrained

hardware," the native 20b — trained at that size, not surgically cut down to it — is

the better answer on the one hard metric available, and it's still smaller than this

pruned 78.3B. This release exists to document that a simple, recovery-free structural

cut is mechanically sound and preserves core capability, not to claim it beats a

purpose-built smaller model.

Agentic coding benchmark

Frontier tier of an agentic coding suite: 9 tasks, real tool-use loop, graded by a fresh

pytest run in a clean checkout. One run, DGX Spark (GB10), stock llama.cpp

(b2276-22397c31a, no patches), --ctx-size 65536 --parallel 1, per-task wall 3600 s,

output cap 16,384 tokens, temp 1.0 / top-p 0.95, reasoning effort high, no gated backoff.

| | result |

|---|---|

| tasks whose solution passed every test | 9 of 9 \* |

| individual test cases passed | 75 of 75 on the 7 scored in the main run |

| mean output tokens per task | 3,194 |

| total wall time | 0.60 h |

\* Two tasks were flagged by the harness rather than scored. backtracking_regex wrote a

solution that passed 59 of 59 tests and then kept generating past the 16,384-token cap,

which the harness voids on truncation regardless of the test outcome — a deliberately

conservative rule. weighted_interval_scheduling hit a transport abort that produced zero

tokens; rerun three times on the same build it passed 8 of 8 tests every time. Neither

is a wrong answer.

Against stock gpt-oss-20b, same family and same settings

Both models were run on the same day, same harness, same build, same sampling. **Applying the

same asterisk rule to both, each solved 9 of 9** — the 20b's two flagged tasks were likewise

solutions that passed their tests (59 of 59 and 8 of 8) and then ran past the cap. So pass/fail

is a tie and output tokens are the discriminator. The table covers the seven tasks scored

outright in both runs:

| task | gpt-oss-20b | gpt-oss-78b | ratio |

|---|---|---|---|

| btree_insert_delete | 27,900 | 2,366 | 11.8x |

| or_set_crdt | 20,980 | 2,340 | 9.0x |

| txn_kv_store | 16,151 | 2,604 | 6.2x |

| tiered_rate_limiter | 10,541 | 3,664 | 2.9x |

| rate_limiter_fix_and_extend | 5,121 | 2,827 | 1.8x |

| mini_sql_executor | 5,173 | 3,690 | 1.4x |

| lru_ttl_cache | 5,310 | 4,870 | 1.1x |

| total | 91,176 | 22,361 | 4.08x |

The pruned 78b reaches the same results using 4.08x fewer output tokens, and finished

the whole suite faster in wall clock (0.60 h vs 0.76 h) despite being 3.7x the file size —

fewer tokens beat more bandwidth. The direction is consistent across all seven tasks and

exceeds 6x on three of them.

Caveat: one run per model. This suite has large run-to-run spread on token counts, so treat

the 4x as a direction with consistent per-task support rather than a precise constant.

It runs on unmodified upstream llama.cpp

Uniform pruning removes experts but leaves the architecture intact, so the file is still

gpt-oss with a smaller expert_count. Verified by running it: stock mainline loaded and

served the whole benchmark with no patch, no fork, no custom kernel. That is the practical

advantage over heterogeneous approaches such as NVIDIA's Puzzle NAS, which varies expert

count and width per layer and needs runtime support that upstream does not have.

Why an APEX quant of gpt-oss-120b doesn't make sense

APEX-style quantization (used elsewhere in this account's releases) works by measuring

per-tensor sensitivity and then non-uniformly reallocating precision — protecting

sensitive tensors, sacrificing bits on insensitive ones — to beat a naive uniform

quantization at the same total size. That has real headroom to exploit when the starting

point is a uniform-precision baseline (e.g. a stock bf16 or Q4_K_M conversion).

gpt-oss-120b doesn't offer that headroom, for two concrete reasons:

1. It already ships pre-allocated, not uniform. OpenAI's own release explicitly

excludes attention, router, embeddings, and lm_head from MXFP4 quantization,

keeping them at full bf16 while only the expert FFN weights go to ~4.25-bit MXFP4.

That's already a "protect the sensitive stuff, compress the rest" allocation baked

into the format — there's very little left for a second, independent

sensitivity-driven pass to discriminate on top of.

1. There's no room to go lower, bit-wise. MXFP4 is already near the practical

floor for this architecture; you can't meaningfully requantize an already-4-bit format

down further and expect a size win the way you can starting from bf16/Q8. The only way

to meaningfully shrink gpt-oss-120b further is structural — remove whole

experts or layers — which is precisely what this experiment does instead. APEX and

structural pruning are answers to different problems: APEX helps when the bit budget

itself is the lever; here, the bit width is already fixed near its floor, and the

count of things resident is the only lever left.

There's also a practical blocker: doing our own from-scratch sensitivity-driven

requantization would require dequantizing the model to a continuous-precision baseline

first, which at 120B scale needs ~240GB just to hold the weights — not something this

was built to attempt, and even if it were, it would at best reproduce what OpenAI's own

MXFP4 calibration already achieved, with no guaranteed improvement.

Files

• gpt-oss-78b.gguf — the pruned checkpoint, converted via mainline

llama.cpp, MXFP4-native experts preserved (not dequantized), bf16 for

attention/router/norms — same mixed-precision layout as the original release.

• kept_expert_indices.json — the exact per-layer set of retained expert indices

(85 of 128, by original index), for reproducibility.

Attribution

• Base model: OpenAI — openai/gpt-oss-120b.
• Reference for the heterogeneous, distillation-recovered approach this experiment

deliberately simplifies away from: NVIDIA —

nvidia/gpt-oss-puzzle-88B.

• Quantization/inference engine: llama.cpp (ggml-org).

Unofficial community experiment; not affiliated with or endorsed by OpenAI or NVIDIA.