---

license: apache-2.0

base_model: tencent/Hy-Memory-30B-A3B

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

language:

- en

- zh

tags:

- gguf

- llama-cpp

- hy-memory

- memory-extraction

- moe

inference: false

---

Hy-Memory-30B-A3B — GGUF

Community GGUF quantizations of Tencent Hy-Memory-30B-A3B, a 30B-parameter MoE memory-writing model with approximately 3B active parameters. Original model by Tencent Hunyuan; conversion and quantization by rapatsky. No additional training was performed.

The conversion corrects an expert-width mismatch in the pinned publisher configuration before conversion. See the reproducibility section below.

Files

| Quantization | Size, GB (decimal) | Size, GiB | Effective file bits/parameter |

| --- | ---: | ---: | ---: |

| IQ4_XS — recommended (separate repository) | 16.907 | 15.746 | 4.499 |

| Q4_K_M | 18.237 | 16.984 | 4.85 |

| Q5_K_M | 21.374 | 19.906 | 5.69 |

| Q6_K | 24.707 | 23.011 | 6.57 |

| Q8_0 | 31.986 | 29.789 | 8.51 |

Each GGUF is a complete, independent model. Download only the quantization you need. The four standard quantizations hosted in this repository use no importance matrix; their router weights and expert-selection biases remain F32. K_M presets use mixed tensor precision, so the labels are not uniform bit widths. The recommended IQ4_XS model is hosted separately and uses RU/EN calibration and a custom tensor allocation, detailed in its model card.

Recommended model: IQ4_XS

IQ4_XS is the recommended default for the locally tested RU/EN memory-extraction workload on Apple M4 Max, replacing the earlier Q5_K_M recommendation. It is 16.907 GB versus Q5_K_M's 21.374 GB, with 4.499 effective whole-file bits/weight. This is an IQ4_XS-based custom mixture of IQ4_XS, Q4_K, Q5_K, Q6_K, Q8_0 and F32, rather than a uniform 4-bit model or an unmodified IQ4_XS preset.

The recommendation balances memory and observed latency with quality. In a separate confirmation set of 128 dialogues, excluded from calibration and candidate selection, the primary failure counts were:

| Model | Russian dialogues with primary failures / 64 | English dialogues with primary failures / 64 |

| --- | ---: | ---: |

| IQ4_XS | 4 | 7 |

| Q5_K_M | 7 | 7 |

| Q8_0 | 4 | 6 |

Primary failures include fact distortions, required-fact omissions and invalid output structure. IQ4_XS introduced failures on 2 Russian and 1 English dialogue relative to each reference, while fixing 5 RU/1 EN failing cases versus Q5 and 2 RU/0 EN versus Q8. One English IQ4_XS response contained invalid JSON and was retained without repair/retry. Equal totals do not imply the same errors or lossless quantization. The source-grounded review was blinded until completion and performed by a single AI agent; this small designed set does not establish population accuracy, independent human evaluation or statistical equivalence. Q5_K_M remains an alternative when avoiding these particular IQ4_XS regressions matters more than memory and latency; it has other failures of its own. Q6_K and Q8_0 provide higher weight precision, but higher precision does not guarantee correct extraction.

Two local timing passes on M4 Max with a 40-core GPU and 64 GiB unified memory observed 10.61–15.44% less complete HTTP-to-EOS time in RU and 8.44–11.75% in EN versus Q5_K_M. These ranges describe the two observed pass means, not confidence intervals or a guaranteed improvement. The separate three-model comparison failed its overall stability gate because Q8 RU time drift was +5.58%, above the predefined 5% threshold. See the IQ4_XS model card for both passes, intervals, full conditions and limitations. These short extraction measurements do not validate long-context performance.

Run with llama.cpp

Tested with llama.cpp revision f3363593f457169cdfbdfe22ea399b1cc139ba18. Use a build with HY V3 (hy_v3) architecture support.

hf download rapatsky/Hy-Memory-30B-A3B-IQ4_XS-GGUF \
  Hy-Memory-30B-A3B-IQ4_XS.gguf chat_template.jinja --local-dir ./hy-memory-gguf

llama-server \
  --model ./hy-memory-gguf/Hy-Memory-30B-A3B-IQ4_XS.gguf \
  --alias hy-memory --host 127.0.0.1 --port 8081 \
  --n-gpu-layers all --flash-attn on \
  --threads 1 --threads-batch 12 --load-mode mmap --cache-ram 0 \
  --ctx-size 32768 --parallel 1 \
  --batch-size 4096 --ubatch-size 1024 \
  --cache-type-k f16 --cache-type-v f16 \
  --jinja --chat-template-file ./hy-memory-gguf/chat_template.jinja \
  --chat-template-kwargs '{"reasoning_effort":"no_think"}'

The command above uses the configuration measured for IQ4_XS on Apple M4 Max. The IQ4_XS card documents the full setup. Results vary with host state and workload; these settings are not a universal optimum. The 32K runtime context above is a tested allocation, not the model's advertised maximum context. To use one of the four standard quantizations, download its GGUF and template from this repository and change --model accordingly.

The original template is embedded in every GGUF; an unchanged standalone copy is included for explicit template selection. reasoning_effort=no_think matches the upstream memory-extraction example. Send greedy extraction requests with temperature: 0 and an appropriate max_tokens limit.

The source checkpoint declares num_nextn_predict_layers=0 and contains no MTP head weights. These GGUF files do not add MTP heads.

Conversion and reproducibility

• Source revision: 64a460a94d5498b7316fb02395c72d01c47a7347.
• Publisher config: expert_hidden_dim=768, but moe_intermediate_size=1536.
• Routed expert and shared expert gate_proj.weight shapes in all 47 MoE layers: [768, 2048]. The local conversion config changed only moe_intermediate_size from 1536 to 768 after checking these shapes.
• HF weights → BF16 GGUF → each of the four quantizations using the same llama.cpp revision. The tokenizer, original chat template and model weights received no training changes.
• Converted metadata: 48 blocks, 128 routed experts, top-8 routing, one shared expert, routed/shared expert width 768, 766 GGUF tensors.
• Conversion environment: Python 3.12, Transformers 5.6.0, PyTorch 2.11.0, safetensors. Transformers 5.6.0 was needed to load the publisher's tokenizer class with this converter.

After building the pinned llama.cpp revision and installing its conversion requirements plus transformers==5.6.0:

hf download tencent/Hy-Memory-30B-A3B \
  --revision 64a460a94d5498b7316fb02395c72d01c47a7347 --local-dir ./hf-model

python conversion/patch_config.py ./hf-model
python llama.cpp/convert_hf_to_gguf.py ./hf-model \
  --outtype bf16 --outfile ./Hy-Memory-30B-A3B-BF16.gguf

for quant in Q4_K_M Q5_K_M Q6_K Q8_0; do
  llama.cpp/build/bin/llama-quantize \
    ./Hy-Memory-30B-A3B-BF16.gguf "./Hy-Memory-30B-A3B-${quant}.gguf" "$quant"
done

The included conversion/patch_config.py is an equivalent reproducibility helper for the validated configuration correction. conversion.json records source and converter revisions, file sizes, tensor metadata and SHA-256 hashes. SHA256SUMS can be used to check downloads of the four standard files here. This standard-preset recipe does not reproduce the custom IQ4_XS model; its separate card and tensor allocation explain the calibration and reproducibility limitations.

Earlier standard-quantization diagnostics and limitations

All four files were loaded and used for inference with full Metal offload on an M4 Max with 64 GB unified memory. A small handcrafted diagnostic comparison used 34 English and 34 Russian dialogues per quantization, greedy decoding, the same prompt/template and FP16 KV cache. Manual semantic grading was blinded to quantization labels.

These earlier diagnostics cover only the four standard files. They use a different dataset and stricter success criterion than the 128-dialogue IQ4_XS confirmation above; do not compare the two tables as one benchmark.

| Quantization | English strict cases | Russian strict cases | Valid JSON |

| --- | ---: | ---: | ---: |

| Q4_K_M | 23/34 | 21/34 | 68/68 |

| Q5_K_M | 23/34 | 22/34 | 68/68 |

| Q6_K | 23/34 | 21/34 | 68/68 |

| Q8_0 | 23/34 | 22/34 | 68/68 |

Strict success required all mandatory facts, correct subject/time/scope, no invented facts, no irrelevant quoted/fictional/question content, deduplication and valid JSON. Q8_0 is a comparison reference, not BF16 ground truth. This is not a representative benchmark or a statistically established quantization ranking; raw diagnostic conversations are not distributed in this model release. Russian was tested locally; upstream language metadata lists English and Chinese.

Shared weaknesses included excess storage, duplication and occasional unsupported inferences. Longer Q5_K_M probes retained the target facts but also produced irrelevant technical notes, a retained earlier city statement, or duplicate project facts. JSON validity does not establish semantic correctness. CPU/server tuning did not fix these model behavior limitations.

Attribution and license

Original model, tokenizer and template: Tencent Hunyuan, upstream model, project. Original copyright and Apache-2.0 license are preserved in LICENSE; conversion changes are documented in NOTICE.

This is a community quantization release and is not an official Tencent release.