---

license: openmdw-1.1

base_model:

- poolside/Laguna-S-2.1

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

tags:

- laguna

- laguna-s-2.1

- gguf

- llama.cpp

- mixture-of-experts

- quantized

- q4-k-m

- q8-0

- dflash

- speculative-decoding

- turing

---

Laguna S 2.1 — attention-Q8 target + Q4_K_M DFlash (GGUF)

Hardware-tuned GGUF pair for running Poolside Laguna S 2.1 on a single 22 GB RTX 2080 Ti with CPU/GPU hybrid offload:

• a layer-aware target derived from Poolside's official imatrix Q4_K_M GGUF, converting only its 240 F16 attention tensors to Q8_0;
• a Q4_K_M conversion of Poolside's official Laguna S 2.1 DFlash draft model.

This is a modified quantized redistribution, not a fine-tune and not an official Poolside release. The target's existing Q4_K, Q6_K, router, norm, and bias tensors were byte-preserved; only the F16 attention tensors were changed.

Files

| File | Size | Tensor layout | Purpose |

|---|---:|---|---|

| laguna-s-2.1-attnQ8.gguf | 67.6 GiB | 239 Q4_K, 48 Q6_K, 240 Q8_0, 287 F32 | Layer-aware target |

| laguna-s-2.1-DFlash-Q4_K_M.gguf | 622 MiB | 45 Q4_K, 4 Q6_K, 27 F32 | Speculative-decoding draft |

The original official target GGUF is about 75 GB. Converting its 240 F16 attention tensors to Q8_0 reduced tensor payload by 2,506 MiB while preserving every already-quantized tensor without requantization.

Why this layout

Laguna S 2.1 is a 118B-A8B MoE. Its official Q4_K_M GGUF deliberately leaves attention in F16, while experts use Q4_K/Q6_K and sensitive router/norm/bias tensors remain F32. On a 22 GB Turing card, target weights are mostly CPU-resident, so reducing attention traffic lets llama.cpp fit more target data on GPU.

The target conversion was intentionally narrow:

• F16 attention → Q8_0
• Q4_K / Q6_K / F32 → COPY unchanged
• no expert-layer requantization
• no router, norm, or bias quantization

RTX 2080 Ti results

AI Lab test system: EPYC 7402, 128 GB eight-channel DDR4 ECC, one modified RTX 2080 Ti 22 GB for Laguna, Poolside's laguna llama.cpp branch, 262K context, one slot.

Target-only decode

| Prompt language | Official Q4_K_M | attention-Q8 | Change |

|---|---:|---:|---:|

| English | 19.640 tok/s | 21.048 tok/s | +7.2% |

| Traditional Chinese | 19.733 tok/s | 21.259 tok/s | +7.7% |

With Q4_K_M DFlash

| Prompt language | Official target + Q4 draft | attention-Q8 + Q4 draft | Change |

|---|---:|---:|---:|

| English | 27.812 tok/s | 28.959 tok/s | +4.1% |

| Traditional Chinese | 18.906 tok/s | 19.498 tok/s | +3.1% |

Strict JSON, native tool-call safety, and executable min_window coding gates passed after target quantization. Laguna's known failure mode—spending the full completion budget in reasoning and returning no final content—was unchanged and should be handled at the agent layer.

Serving with llama.cpp

Use Poolside's llama.cpp fork, laguna branch. Generic upstream llama.cpp does not yet contain the complete Laguna DFlash decoder contract required by this pair.

./build/bin/llama-server \
  -m laguna-s-2.1-attnQ8.gguf \
  -md laguna-s-2.1-DFlash-Q4_K_M.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 4 \
  -ngld 99 \
  -c 262144 \
  -ctk q4_0 -ctv q4_0 \
  -ctkd f16 -ctvd f16 \
  -t 24 -tb 24 -td 12 -tbd 12 \
  -fa on \
  -fit on -fitt 7680 -fitc 262144 \
  --parallel 1 \
  --jinja \
  --reasoning on --reasoning-format deepseek \
  --host 0.0.0.0 --port 8095

Important DFlash caveats

Keep draft KV in F16. Q4_0 draft KV reduced proposal acceptance below 1% and decode to roughly 5–6 tok/s. Q4_K_M draft weights and Q4_0 runtime KV are independent quantizations; their errors do not cancel.

Tune --spec-draft-n-max for the hardware and draft precision. Poolside's official BF16 example uses n=15. With this Q4_K_M draft on the tested Turing hybrid-offload system, n=4 was the best measured point. Larger values reduced acceptance enough to lose throughput.

The target KV can remain Q4_0; the precision warning applies specifically to the DFlash draft K/V cache (-ctkd and -ctvd).

Context and reasoning

This package was validated at 262,144 tokens. The native target was trained up to 1M context; see Poolside's official model card for the 1M YaRN override and quality caveats.

For tool/coding action lanes, consider disabling thinking or adding an empty-final repair. A short max_tokens cap alone can consume the whole budget inside reasoning_content before a final answer is emitted.

Credits

• Base model and architecture: poolside/Laguna-S-2.1
• Official GGUF and importance matrix: poolside/Laguna-S-2.1-GGUF
• DFlash draft: poolside/Laguna-S-2.1-DFlash
• Layer-aware target conversion, Q4_K_M draft conversion, Turing tuning, and validation: coolthor

License

Distributed under the OpenMDW-1.1 License, inherited from Laguna S 2.1. A complete copy is included as LICENSE.md. The files in this repository were modified through quantization; Poolside does not endorse or maintain this redistribution.

---

☕ If this saved you GPU hours, you can buy me a coffee.