---

license: apache-2.0

language:

• ja
• en

base_model: elyza/ELYZA-Thinking-1.0-llm-jp-4-32b-a3b

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

• gguf
• llama.cpp
• llm-jp-4
• reasoning
• imatrix

---

ELYZA-Thinking-1.0-llm-jp-4-32b-a3b-GGUF

GGUF quantizations of elyza/ELYZA-Thinking-1.0-llm-jp-4-32b-a3b.

> [!IMPORTANT]

> Stock llama.cpp cannot chat with this model yet: /v1/chat/completions fails. Apply scripts/llmjp4-space-after-control.patch to llama.cpp and rebuild. See NOTES.md for why.

> [!TIP]

> The model is sensitive to long runs of blank lines in the prompt (for example a raw terminal screen). If you pass such text, collapse runs of 3+ newlines first. See NOTES.md.

Files

| File | Size | KLD (ja) | KLD (en) | Notes |

|---|---|---|---|---|

| BF16/…-BF16-0000{1,2}-of-00002.gguf | 64.3 GB | – | – | Split in 2 |

| …-Q8_0.gguf | 34.2 GB | 0.0096 | 0.0143 | |

| …-Q6_K.gguf | 28.8 GB | 0.0155 | 0.0250 | |

| …-Q5_K_M.gguf | 24.4 GB | 0.0213 | 0.0350 | |

| …-Q4_K_M.gguf | 21.4 GB | 0.0383 | 0.0669 | |

| …-IQ4_XS-dyn.gguf | 20.5 GB | 0.0208 | 0.0407 | Per-tensor mix. Recommended for ~24 GB of memory |

| …-IQ4_XS.gguf | 17.6 GB | 0.0439 | 0.0747 | |

| …-Q3_K-dyn.gguf | 17.8 GB | 0.0476 | 0.1004 | Per-tensor mix. Not recommended: IQ4_XS is better at the same size |

| …-Q3_K_M.gguf | 16.7 GB | 0.0889 | 0.1559 | |

Quantized with an importance matrix (imatrix.gguf). KLD is the mean KL divergence from BF16 on Japanese and English Wikipedia text; lower is closer to the original. -dyn files keep attention and output tensors at higher precision and spend the low bits on the MoE experts.

Usage

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git apply /path/to/llmjp4-space-after-control.patch
cmake -B build -DGGML_VULKAN=ON   # or -DGGML_CUDA=ON, etc.
cmake --build build -j --target llama-server

./build/bin/llama-server -m ELYZA-Thinking-1.0-llm-jp-4-32b-a3b-IQ4_XS-dyn.gguf \
  --jinja -ngl 99 -c 65536 -fa on --temp 0.6 --top-p 0.95

The patch was tested on llama.cpp 4ebdf2c. Recommended sampling is the same as the original model: temperature=0.6, top_p=0.95.

License and credits

Apache License 2.0, the same as the original model. LICENSE and NOTICE are included.

• Original model: elyza/ELYZA-Thinking-1.0-llm-jp-4-32b-a3b (ELYZA, Inc.)
• Base model: llm-jp/llm-jp-4-32b-a3b-base (LLM-jp)
• imatrix data: TFMC/imatrix-dataset-for-japanese-llm (ODC-BY) and bartowski's calibration_datav3