---

license: apache-2.0

base_model: unsloth/Qwen3.8-Flash-Next-GGUF

library_name: gguf

pipeline_tag: text-generation

tags:

- qwen

- qwen4exp

- gguf

- llama.cpp

- moe

- quantized

- ultra-low-bit

- text-generation

---

⚡ Qwen3.8 Flash Next 125B — UltraLite 37 GiB GGUF

> 125B parameters. 37 GiB. One wild GGUF.

This experimental text-only build compresses Qwen3.8-Flash-Next from a

72.55 GB calibrated source GGUF to 39,721,239,200 bytes:

• 36.993287 GiB
• 39.721239 decimal GB
• 1.80 BPW tensor payload
• 125B total parameters / approximately 6B active parameters
• 278,760,800 bytes below a strict 40,000,000,000-byte ceiling

The release is not a blind whole-model requant. It uses a patched GGUF layout,

streaming row repacking, mixed quantization by tensor role, boundary-layer

protection, byte-for-byte preservation of sensitive paths, and structural

removal of optional components not used by ordinary text generation.

Follow the experiments

If you want more aggressively compressed models, weird inference experiments,

and build notes, follow @procrastiness on X.

Important runtime requirement

This model uses a padded physical representation for

per_layer_token_embd.weight. It therefore requires the included

patches/qwen4exp-under40.patch applied to the pinned Qwen4Exp llama.cpp

revision. Stock llama.cpp does not understand this custom physical/logical row

layout yet.

The patch is part of the release, and the exact build/runtime revision is pinned

in build-manifest.txt.

What was compressed

The source is Unsloth's UD-IQ1_S release at revision

83cadfda58d30be06c110518208d1bb918b33f10. Its three source shards total

72,546,461,344 bytes, with 72,535,436,800 bytes of tensor data.

| Tensor group | Source payload | Source format | UltraLite format | Reasoning |

|---|---:|---|---|---|

| 320,001,536 × 160 per-layer token table | 28,800,138,240 B | IQ4_NL | padded Q1_0 | Largest single tensor; custom streaming repack makes the target size possible |

| 48 MoE down-projection banks | 22,649,241,600 B | IQ4_NL | Q2_0 | Down projections are retained at two-bit precision across every layer |

| 96 MoE gate/up banks | 17,196,646,400 B | IQ1_S / IQ2_XXS | Q1_0 interior, Q2_0 boundaries | Maximum savings in redundant expert interiors while protecting both ends of the network |

| Attention, routers, shared experts, hyper-connections, norms, SSM, indexers, token/output embeddings | 3,889,410,560 B | BF16/F32/Q8/Q6/Q5/Q4 mix | copied unchanged | Keeps sensitive dense and routing paths identical to the calibrated source |

Boundary-layer protection

Expert gate and up projections use Q2_0 in layers:

0, 1, 2, 44, 45, 46, 47

Layers 3–43 use Q1_0 for those expert banks. All 48 expert down banks remain

Q2_0. This spends the final byte budget on the transformer entrance and exit,

where error propagation is especially undesirable.

The 160 → 256 padded PLE architecture

The per-layer embedding table contains 320,001,536 logical rows of 160 values.

The target ultra-low-bit block layout cannot directly encode that row width.

The included llama.cpp patch therefore:

1. Reads the source table in bounded chunks of 32,768 rows.
2. Decodes each 160-value logical row.
3. Appends 96 zeros, producing a 256-value physical row.
4. Quantizes that physical row to Q1_0.
5. Stores the physical width in GGUF tensor metadata.
6. At inference, gathers the padded physical row.
7. Creates a strided view of its first 160 values.
8. Materializes the original logical shape before the normal Qwen4Exp graph

consumes it.

The padding does not prune learned PLE values: every original element remains

present. Only the 96 appended positions are zeros. Quantization error still

exists, as expected from Q1_0.

Structural pruning

• The optional vision projector is not included; this is a text-only release.
• The normal-generation source GGUF does not contain the separate 4B MTP

training/speculation companion head.

• No ordinary text-generation transformer layer was removed.
• No attention, router, shared-expert, normalization, SSM, indexer, token

embedding, or output tensor was pruned.

Exact result

| Artifact | Bytes | SHA-256 |

|---|---:|---|

| Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00001-of-00003.gguf | 10,946,624 | 7d0be62f3335e53f7d83a6f2b903a5619014f5de4f7453f0f2095d6d28fb4dda |

| Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00002-of-00003.gguf | 25,357,596,224 | b7870d03fdadb5acf38cfaaf46015a638713d67af96db2ae3268c4e0b8200e53 |

| Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00003-of-00003.gguf | 14,352,696,352 | 58cd80f5225550df9b0de7b8b6c1299c242774cff798256b5f445ce42a053795 |

| Total | 39,721,239,200 | See SHA256SUMS |

All 1,224 tensors converted successfully. The patched runtime loaded the three

shards, accepted the padded PLE representation, evaluated a prompt, generated a

deterministic first token, and exited with code 0. Smoke-test stderr was empty.

Build method

The reproducible pipeline is included as scripts/remote_build.sh. It performs

the following complete sequence:

1. Pins the source model repository and exact source revision.
2. Downloads all three source shards with resumable parallel transfers.
3. Validates every source shard against its exact expected byte count.
4. Checks out Unsloth's Qwen4Exp llama.cpp branch at commit

250b61446efc91e3a179c8677956f2667c8fbda0.

1. Applies patches/qwen4exp-under40.patch.
2. Builds llama-quantize, llama-cli, and llama-perplexity.
3. Executes an exact dry-run allocation with explicit per-tensor overrides.
4. Refuses conversion unless projected payload plus 32 MiB of GGUF overhead is

below 40,000,000,000 bytes.

1. Streams and requantizes only the selected tensors; unchanged tensors are

copied byte-for-byte.

1. Keeps the source three-shard split.
2. Enforces the hard size limit against final filesystem byte counts.
3. Runs a deterministic non-interactive generation smoke test.
4. Generates SHA-256 checksums and a complete build manifest.

The effective quantization command is:

llama-quantize \
  --allow-requantize \
  --keep-split \
  --tensor-type '^per_layer_token_embd[.]weight$=Q1_0' \
  --tensor-type '^blk[.](0|1|2|44|45|46|47)[.]ffn_(gate|up)_exps[.]weight$=Q2_0' \
  --tensor-type '^blk[.][0-9]+[.]ffn_down_exps[.]weight$=Q2_0' \
  --tensor-type '^blk[.][0-9]+[.]ffn_(gate|up)_exps[.]weight$=Q1_0' \
  SOURCE-00001-of-00003.gguf \
  Qwen3.8-Flash-Next-125B-UltraLite-37GiB \
  COPY \
  "$(nproc)"

COPY is intentionally used as the default. The patch permits explicit tensor

overrides on top of COPY, so everything not matched by the four expressions

retains its original calibrated encoding.

Build it yourself

Required tools on Linux:

sudo apt-get update
sudo apt-get install -y git git-lfs cmake build-essential curl aria2

Then place the patch beside the script and run:

mkdir -p /root/qwen38-compress
cp scripts/remote_build.sh /root/qwen38-compress/
cp patches/qwen4exp-under40.patch /root/qwen38-compress/
chmod +x /root/qwen38-compress/remote_build.sh
/root/qwen38-compress/remote_build.sh

The process needs roughly 73 GB for the source, 40 GB for the output, build

space, and temporary headroom. The custom PLE conversion is chunked so it does

not allocate the full decoded table in memory.

Run the model

Clone and patch the pinned runtime:

git clone --branch qwen4exp/qwen3.8-flash-next \
  https://github.com/unslothai/llama.cpp.git
cd llama.cpp
git checkout 250b61446efc91e3a179c8677956f2667c8fbda0
git apply ../patches/qwen4exp-under40.patch

cmake -S . -B build -DGGML_CUDA=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j "$(nproc)" --target llama-cli

Run from the directory containing all three GGUF shards:

./llama.cpp/build/bin/llama-cli \
  -m Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00001-of-00003.gguf \
  -c 4096 \
  -t "$(nproc)"

The build was validated CPU-only. GPU backends can be added to the patched

runtime in the normal llama.cpp build configuration when supported by the host.

Perplexity comparison

Use the same corpus, context, batch sizes, and runtime for both source and

UltraLite:

LLAMA=./llama.cpp/build/bin/llama-perplexity
SRC=/path/to/Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf
DST=/path/to/Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00001-of-00003.gguf
CORPUS=/path/to/calibration.txt

"$LLAMA" -m "$SRC" -f "$CORPUS" -c 2048 -b 128 -ub 128 -t "$(nproc)"
"$LLAMA" -m "$DST" -f "$CORPUS" -c 2048 -b 128 -ub 128 -t "$(nproc)"

Reproducibility files

• build-manifest.txt — exact sizes, revisions, format choices, smoke result,

and checksums

• SHA256SUMS — shard integrity hashes
• patches/qwen4exp-under40.patch — converter and runtime support
• scripts/remote_build.sh — complete resumable build pipeline
• scripts/deploy_to_triologay.ps1 — Tailscale deployment helper used for the

original build

Credits

• Qwen for the model family
• Unsloth's Qwen3.8-Flash-Next GGUF release
• llama.cpp
• Qwen4Exp llama.cpp work

---

Built as an aggressive compression experiment. For the next one, follow

@procrastiness on X.