---

license: other

license_name: openmdw-1.1

license_link: https://huggingface.co/poolside/Laguna-S-2.1/blob/main/LICENSE.md

base_model: poolside/Laguna-S-2.1

base_model_relation: quantized

quantized_by: vcruz305

pipeline_tag: text-generation

library_name: gguf

tags:

• gguf
• imatrix
• laguna
• moe
• quantized

---

Laguna-S-2.1 — GGUF quant suite (1–8 bit, imatrix)

Community GGUF quantizations of

poolside/Laguna-S-2.1,

covering the low-bit band (IQ1–IQ3) that the official release does not

ship — so you can run this 117B-parameter MoE on a single GPU.

Every file is quantized with poolside's official importance matrix

(laguna-s-2.1.imatrix, 400 chunks) from their official F16 GGUF, using the

laguna architecture in llama.cpp. Each file is **load-and-generate smoke

tested on GPU before upload* (see Verification* below).

> Uploading now, smallest first. Files appear as they finish. This card

> lists the full planned suite; the Status column shows what is live.

---

Model at a glance

| | |

|---|---|

| Architecture | laguna (poolside) — Mixture-of-Experts |

| Total parameters | ~117B (256 × 4.5B label) |

| Experts | 256 routed, top-10 active per token |

| Layers | 48 |

| Hidden size | 3072 |

| Attention | hybrid full / sliding-window (512), GQA (48 Q / 8 KV heads, head_dim 128) |

| Context | 262,144 native · up to 1,048,576 via YaRN |

| Base precision | bfloat16 |

Because only ~10 of 256 experts fire per token, decode is far faster than the

117B total suggests — this is what makes low-bit single-GPU serving practical.

---

Available quants

Recommended picks in bold. Sizes are file sizes; add KV cache + compute

buffers for total VRAM (see Hardware).

| Quant | Size | Status | Notes |

|---|---|---|---|

| IQ1_S | 23.8 GB | ✅ live | bonus: absolute smallest; single 24 GB GPU |

| IQ1_M | 26.4 GB | ✅ live | recommended smallest; better quality than IQ1_S |

| IQ2_M | ~40 GB | ⏳ uploading | best quality/size in the 2-bit band |

| Q2_K | ~40 GB | ⏳ | |

| IQ3_XXS | ~45 GB | ⏳ | split |

| Q3_K_M | ~56 GB | ⏳ | split; strong mid-range |

| IQ4_XS | ~63 GB | ⏳ | split; near-original quality |

| Q4_K_M | ~67 GB | ⏳ | split |

| Q5_K_M | ~83 GB | ⏳ | split |

| Q8_0 | ~128 GB | ⏳ | split; near-lossless |

For F16, use poolside's official

Laguna-S-2.1-GGUF.

Split files: quants above ~48 GB are split with llama-gguf-split. Download

all parts (*-00001-of-000NN.gguf …) into one folder and point llama.cpp at

part 00001; it auto-merges. No manual concatenation.

---

Hardware / VRAM

Total VRAM ≈ file size + KV cache + ~0.5–1 GB compute buffers.

KV cache for this model is ~192 KiB/token in fp16 (~0.75 GB at 4K, ~1.5 GB

at 8K). Use -fa on and an fp8/q8 KV cache to cut that roughly in half.

| Your GPU(s) | Recommended quant | Context |

|---|---|---|

| single 24 GB (3090/4090/RTX 6000) | IQ1_S | ~4–8K on-GPU; longer via --n-cpu-moe offload |

| single 32 GB | IQ1_M / IQ2_M | comfortable mid context |

| 48 GB (or 2×24) | IQ2_M / IQ3_XXS | long context |

| 64 GB | Q3_K_M / IQ4_XS | long context, near-orig quality |

| 80 GB+ | IQ4_XS and up | full fidelity |

Not enough VRAM? Because it's a sparse MoE, CPU-offloading the expert tensors

is cheap. Add --n-cpu-moe 40 (offload experts of 40 layers to RAM) or classic

-ngl partial offload; you keep most of the speed with a fraction of the VRAM.

---

How to run

> ⚠️ Requires poolside's llama.cpp fork (branch laguna) until upstream

> support lands — see ggml-org/llama.cpp#25165.

> Stock/mainline llama.cpp will error with unknown model architecture: 'laguna'.

1. Build the fork

git clone --branch laguna https://github.com/poolsideai/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release   # drop -DGGML_CUDA=ON for CPU-only
cmake --build build -j

2. Download a quant

pip install -U "huggingface_hub[cli]"
hf download vcruz305/Laguna-S-2.1-GGUF Laguna-S-2.1-IQ1_S.gguf --local-dir ./laguna
# split quant? grab every part:
# hf download vcruz305/Laguna-S-2.1-GGUF --include "Laguna-S-2.1-IQ4_XS-*.gguf" --local-dir ./laguna

3a. Chat (CLI)

./build/bin/llama-cli \
  -m ./laguna/Laguna-S-2.1-IQ1_S.gguf \
  -ngl 999 -fa on -c 8192 --jinja \
  -p "The capital of France is"

3b. OpenAI-compatible server

./build/bin/llama-server \
  -m ./laguna/Laguna-S-2.1-IQ1_S.gguf \
  -ngl 999 -fa on -c 8192 --jinja --port 8000
# then: curl http://localhost:8000/v1/chat/completions ...

Low-VRAM (offload experts to RAM)

./build/bin/llama-cli -m ./laguna/Laguna-S-2.1-IQ1_S.gguf \
  -ngl 999 --n-cpu-moe 40 -fa on -c 8192 --jinja -p "Hello"

Contexts beyond 256K

The weights are native 1M checkpoints. To exceed 256K, override rope/YaRN at

load time per poolside's base card (e.g. `--rope-scaling yarn --yarn-orig-ctx

262144`); most users will not need this.

---

Verification

Each uploaded file was smoke-tested on GPU (offloaded, fresh load) and must

emit the expected continuation before upload — e.g. IQ1_S:

Prompt: "The capital of France is"
Output: "The capital of France is **Paris**."

Files that fail the load-and-generate check are not uploaded. Quant sizes

and short SHA-256 receipts are recorded in each file's commit message.

---

Choosing a quant

• Want it to fit one 24 GB card? → IQ1_S. It's genuinely coherent for a

1.62-bit 117B model; expect some quality loss vs higher bits.

• Best low-bit quality/size? → IQ2_M or Q3_K_M.
• Near-original quality? → IQ4_XS and up (needs 64 GB+ or offload).
• IQ (i-quants) are generally higher quality per byte than the same-size K-quant,

but need the imatrix (already baked in here) and are slightly slower on some

CPUs.

---

Credits & license

• Base model, architecture, and importance matrix:

poolside — please cite/credit them.

• Speculative-decoding draft model for extra speed:

Laguna-S-2.1-DFlash.

• Quantization: vcruz305, llama.cpp (laguna fork).
• Released under OpenMDW-1.1, following the base model's license. See

LICENSE.md.

Not affiliated with poolside. Community redistribution of quantized weights.