---

base_model: tokyotech-llm/Qwen3-Swallow-32B-SFT-v0.2

base_model_relation: quantized

license: apache-2.0

language:

• ja
• en

library_name: llama.cpp

pipeline_tag: text-generation

tags:

• gguf
• quantized
• imatrix
• qwen3
• japanese

---

Qwen3-Swallow-32B-SFT-v0.2 GGUF

GGUF quantizations of tokyotech-llm/Qwen3-Swallow-32B-SFT-v0.2 for llama.cpp-compatible runtimes. This repository contains quantized derivatives of the base model. It is an unofficial community release, not a Swallow team release.

Files and measurements

| Quant | Size | KL mean | KL p99 | Top-1 agreement | Generation | GPU layers |

|---|---:|---:|---:|---:|---:|---:|

| Q8_0 | 34.8 GB | 0.0011 | 0.007 | 98.3% | 3.7 tok/s | 40 |

| Q6_K | 26.9 GB | 0.0027 | 0.019 | 97.4% | 8.0 tok/s | 52 |

| Q5_K_M | 23.2 GB | 0.0049 | 0.044 | 96.7% | 19.8 tok/s | 60 |

| Q4_K_M | 19.8 GB | 0.0115 | 0.122 | 95.7% | 70.7 tok/s | 99 |

| IQ4_XS | 17.7 GB | 0.0180 | 0.192 | 94.6% | 80.3 tok/s | 99 |

| IQ3_M | 14.9 GB | 0.0549 | 0.645 | 91.4% | 84.6 tok/s | 99 |

Q4_K_M is the practical default for a 24 GB GPU. Use a larger quant if memory permits, or IQ3_M if space matters more than fidelity.

KL and top-1 agreement compare each quant with the F16 conversion on disjoint held-out Japanese and English Wikipedia text. Lower KL and higher top-1 agreement indicate closer next-token distributions.

Generation was measured with llama-bench, single stream, on an RTX 5090 with a 24 GB GPU memory budget. GPU layers is the number of layers offloaded for that result. Throughput will vary with hardware, context length, runtime version, and offload settings. See measurements.csv for the full results.

Usage

llama-cli -hf ultimatechris/Qwen3-Swallow-32B-SFT-v0.2-GGUF:Q4_K_M -ngl 99 -c 4096 -cnv

ollama run hf.co/ultimatechris/Qwen3-Swallow-32B-SFT-v0.2-GGUF:Q4_K_M

日本語

Swallow の日本語向け 32B モデルを llama.cpp 用 GGUF に量子化したものです。

24 GB の GPU では Q4_K_M が標準的な選択です。より小さいファイルが必要なら IQ3_M、忠実度を優先するなら Q5_K_M 以上を選んでください。

表の KL と top-1 一致率は F16 版との出力差、tok/s は RTX 5090 を 24 GB に制限した測定値です。実際の速度は GPU、CPU、コンテキスト長、GPU offload 設定で変わります。

ollama run hf.co/ultimatechris/Qwen3-Swallow-32B-SFT-v0.2-GGUF:Q4_K_M

Method and limitations

The F16 model was converted and quantized with llama.cpp. The importance matrix was built from Japanese and English Wikipedia text. The evaluation set uses different source samples.

These files preserve the base model's capabilities and limitations. Quantization can change outputs, especially at lower bitrates. The measurements above are distribution comparisons, not task benchmarks. Read the base model card before deployment.