Laguna-S-2.1 UD-IQ4_S GGUF

<!-- polished-overview:start -->

Overview

Laguna-S-2.1-UD-IQ4 S-GGUF is a GGUF release for llama.cpp-compatible runtimes and local inference, published by groxaxo.

It is intended for open-source evaluation, reproducible experimentation, and compatible local or

hosted inference workflows. The wording below is deliberately limited to what can be verified

from this repository's metadata and artifacts.

At a glance

| Field | Details |

|---|---|

| Format | GGUF |

| Source / base | poolside/Laguna-S-2.1 |

| Intended task | the task described by the included configuration and documentation |

| License | the license declared in the repository files |

What is included

• *.gguf (5 files)
• Additional configuration, tokenizer, processor, or shard files (5 visible artifacts total)

Quick start

llama.cpp

Download a .gguf file that fits your available memory, then run it with a current llama.cpp

build:

llama-cli \
  -m /path/to/model.gguf \
  -p "Write a concise technical summary."

For vision or any-to-any models, download the matching multimodal projection file when one is

provided and follow the source model's modality-specific instructions.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for

testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

<!-- polished-overview:end -->

Custom Unsloth Dynamic quantization of Poolside Laguna-S-2.1,

optimized for 3× RTX 3090 (24 GB VRAM each) inference with llama.cpp.

Why this quant?

This is an intermediate recipe between Unsloth's UD-Q4_K_S and UD-IQ4_NL,

tuned specifically for 3× 3090 setups where:

• UD-Q4_K_S (Q6_K experts) → too large, doesn't fit comfortably in 72 GB total VRAM
• UD-IQ4_NL (IQ2_S experts) → fits but quality degraded on expert routing
• UD-IQ4_S (this) → sweet spot: fits 72 GB with headroom, retains expert quality

Quantization recipe

Built with llama-quantize using Unsloth's importance matrix (imatrix_unsloth.gguf)

and a per-tensor type override file.

| Precision | Tensors | Assignment |

|------------|---------|------------|

| Q2_K | 386 | Base experts + attention (shared with UD-Q4_K_S) |

| F32 | 287 | Norms, scales, gate inputs |

| Q5_K | 69 | ffn_down_exps (all 45) + ffn_up_exps layers 1-24 |

| IQ4_XS | 70 | ffn_gate_exps (all 47) + ffn_up_exps layers 25-47 |

| IQ2_XXS | 2 | blk.46.ffn_down_exps + blk.47.ffn_down_exps (last layers) |

| Total | 814 | 256×4.5B MoE, 48 layers |

BPW: 4.76 | Size: ~67 GB | Compression: 3.3× from BF16

Logic

• ffn_down_exps (output projection, most quality-sensitive) → Q5_K (higher precision)
• ffn_gate_exps (gating, least sensitive) → IQ4_XS (more compressed)
• ffn_up_exps split by layer depth — early layers (1-24) get Q5_K, late layers (25-47) get IQ4_XS
• Last 2 layers' ffn_down_exps → IQ2_XXS (minimal quality impact, saves space)
• Uses Unsloth's official imatrix computed on unsloth_calibration_Laguna-S-2.1.txt

3× RTX 3090 usage

llama-server \
  -m Laguna-S-2.1-UD-IQ4_S-00001-of-00005.gguf \
  -ngl 999 \
  -ts 1,1,1 \
  -c 8192 \
  --host 0.0.0.0 --port 8080

All 48 layers + 256 experts offload to GPU. Fits comfortably in 72 GB total VRAM

with room for context.

Source

• Base model: unsloth/Laguna-S-2.1-GGUF (BF16)
• Imatrix: imatrix_unsloth.gguf from the same repo
• Quantization tool: poolside-llama.cpp-laguna (custom llama.cpp fork with Laguna arch support)

License

OpenMDW 1.1 (same as base model)