---

license: apache-2.0

base_model: tencent/Hy-Memory-30B-A3B

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

language:

- en

- zh

tags:

- gguf

- llama-cpp

- hy-memory

- memory-extraction

- moe

- iq4_xs

- mixed-precision

inference: false

---

Hy-Memory-30B-A3B — IQ4_XS

Community quantization of Tencent Hy-Memory-30B-A3B, a memory-writing MoE model with approximately 30B total and 3B active parameters. Original model by Tencent Hunyuan; conversion and custom quantization by rapatsky.

Quantization: IQ4_XS-based mixed precision with a custom tensor allocation, 4.4989 effective bits per weight. This is a complete GGUF model for llama.cpp. It uses native IQ4_XS, Q4_K, Q5_K, Q6_K, Q8_0 and F32 tensors. It is not MXFP4 or the standard Q4_K_M preset. Running a standard IQ4_XS quantization command alone will not reproduce this allocation.

Recommended default for the locally tested RU/EN extraction workload on M4 Max. See the main GGUF repository for the other available quantizations. The recommendation balances memory and observed latency with quality; the known Q5/Q8 regressions are documented below.

Download

| File | Bytes | Size, decimal GB | Effective file bits/weight |

| --- | ---: | ---: | ---: |

| Hy-Memory-30B-A3B-IQ4_XS.gguf | 16,907,329,280 | 16.907 | 4.4989145999 |

Effective bits/weight here means the entire GGUF file size × 8 / 30,064,725,888 parameters, including metadata, quantization scales and alignment. It is not a uniform 4-bit encoding, nor a runtime RAM estimate. KV cache and runtime buffers require additional memory.

hf download rapatsky/Hy-Memory-30B-A3B-IQ4_XS-GGUF \
  Hy-Memory-30B-A3B-IQ4_XS.gguf chat_template.jinja \
  --local-dir ./hy-memory-iq4-xs

SHA256SUMS covers every release file other than itself. The GGUF SHA-256 is:

ccac4408c5a85e8ec6c4804fd0b5ce231f83e515b0f3b35e0121e6185f919b2f

Tensor allocation

The GGUF has 766 tensors. Parameter shares refer to all model parameters, not tensor counts.

| Stored type | Tensors | Parameters | Share of parameters |

| --- | ---: | ---: | ---: |

| IQ4_XS | 132 | 26,575,110,144 | 88.39299% |

| Q5_K | 8 | 1,610,612,736 | 5.35715% |

| Q6_K | 96 | 1,055,916,032 | 3.51214% |

| Q8_0 | 242 | 609,222,656 | 2.02637% |

| Q4_K | 1 | 201,326,592 | 0.66964% |

| F32 | 287 | 12,537,728 | 0.04170% |

Most routed-expert parameters use IQ4_XS. Shared experts, the output projection and attention K/V use Q8_0; embeddings use Q6_K. Norms and routing tensors retain their original F32 representation. The exact allocation is included in tensor-types.txt, and quantization.json records counts, provenance and checksums. The file-level GGUF type is MOSTLY_IQ4_XS (30); the tensor table specifies the actual mixture.

This custom allocation includes a three-matrix change relative to the intermediate out8_emb6 allocation: the first dense FFN up projection, blk.0.ffn_up.weight, was raised from Q6_K to Q8_0; attention-Q matrices in blocks 13 and 21 were lowered from Q8_0 to Q6_K to remain within the 4.5 bits/weight budget.

Run with llama.cpp

Tested with llama.cpp revision f3363593f457169cdfbdfe22ea399b1cc139ba18, with HY V3 (hy_v3) support and the Metal backend. The following is the locally used single-request configuration on Apple M4 Max, 40-core GPU, 64 GiB unified memory:

llama-server \
  --model ./hy-memory-iq4-xs/Hy-Memory-30B-A3B-IQ4_XS.gguf \
  --alias hy-memory --host 127.0.0.1 --port 8081 \
  --n-gpu-layers all --flash-attn on \
  --threads 1 --threads-batch 12 --load-mode mmap --cache-ram 0 \
  --ctx-size 32768 --parallel 1 \
  --batch-size 4096 --ubatch-size 1024 \
  --cache-type-k f16 --cache-type-v f16 \
  --jinja --chat-template-file ./hy-memory-iq4-xs/chat_template.jinja \
  --chat-template-kwargs '{"reasoning_effort":"no_think"}'

The unchanged upstream chat template is also embedded in the GGUF. Use temperature: 0 for deterministic extraction and an appropriate output token limit. An example system instruction is: Extract explicit user facts and preferences. Write concise, self-contained memories. Do not infer or invent information. Return only a JSON object with a memories list of strings. See the upstream usage example for the original API request.

The 32K context is the tested runtime allocation; the measurements below do not validate quality or speed at the advertised maximum context. No MTP/EAGLE draft head or speculative decoding is included. Other engines and hardware have not been qualified for this release.

Conversion and calibration

• Source: tencent/Hy-Memory-30B-A3B at 64a460a94d5498b7316fb02395c72d01c47a7347.
• The publisher configuration had expert_hidden_dim=768 but moe_intermediate_size=1536. Before BF16 GGUF conversion, only moe_intermediate_size was corrected to 768, matching the actual routed/shared expert tensor width. The resulting architecture has 48 blocks, 128 routed experts, top-8 routing, and one shared expert in each MoE block.
• Quantized tensor payloads were produced directly from the BF16 GGUF representation. The final mixture copies native quantized payloads exactly; it does not requantize an already quantized checkpoint. Original F32 tensors are preserved. No additional model training was performed.
• For publication, only two informational quantize.imatrix metadata values were shortened from absolute local paths to file basenames. All 766 tensor payloads and all other metadata, including runtime/tokenizer configuration, were verified byte-for-byte/equivalent against the locally qualified model. This changes the whole-file checksum, not model weights or inference settings. The size figures and checksum above describe the published file.
• Importance statistics came from Q8 activation moments on a calibration split of 128 Russian and 128 English training dialogues. Embeddings used an explicitly synthetic unit-weight importance fallback, not measured embedding activation statistics. Allocation considered weighted quantization error and subsequent dev/selection measurements under an actual file-size budget.
• The candidate was selected among 23 frozen variants using 500 selection dialogues. The separate 128-dialogue confirmation set was excluded from calibration, allocation and candidate selection; its results did not trigger recipe or prompt changes.

The tensor allocation is published, but calibration texts, the importance matrix and raw evaluation conversations are not included. Exact recalibration is therefore not reproducible from this repository alone. BF16 was used as the weight source, not as an inference reference; Q8 was the numerical comparison reference. Q8 output is not factual ground truth.

Local quality check: Russian and English

After freezing the candidate, 128 additional dialogues (64 per language) were evaluated with the same prompts and greedy decoding for IQ4_XS, Q5_K_M and Q8_0. Source-grounded semantic review was blinded to model identity until completion and performed by a single AI agent, not independent human graders. These are local, designed test cases and do not establish population accuracy or statistical equivalence.

| Model | Russian dialogues with primary failures / 64 | English dialogues with primary failures / 64 |

| --- | ---: | ---: |

| IQ4_XS | 4 | 7 |

| Q5_K_M | 7 | 7 |

| Q8_0 | 4 | 6 |

Primary failures include fact distortions, omissions and invalid output structure. Relative to each reference, IQ4_XS introduced failures on 2 Russian and 1 English dialogue. It also fixed 5 RU/1 EN failing cases versus Q5 and 2 RU/0 EN versus Q8. One English IQ4_XS response had invalid JSON; it was retained without repair/retry and is included in the table. Additional erroneous assertions inside already-failing dialogues were tracked separately; dialogue totals do not enumerate every error.

Equal aggregate counts do not mean identical answers or no regressions. This release does not claim lossless quantization, zero new errors, proven equivalence to Q8/BF16, or a globally optimal allocation. Upstream language metadata lists English and Chinese; Russian was tested locally, while Chinese was not evaluated in this quantization study.

Local speed observations

On the M4 Max configuration above, 48 fixed training-split dialogues (24 per language) were measured with two repeats per model, cold KV cache, temperature=0, seed 42, and an output limit of 4096 tokens. All measured requests reached EOS. Times cover the complete resident-model HTTP request to EOS, excluding model loading; these are short extraction outputs, not fixed-token throughput tests. The machine was on AC with reported power mode 2, not Low Power mode.

| Comparison pass | Language | Q5_K_M mean time, s | Q8_0 mean time, s | IQ4_XS mean time, s | Less time vs Q5 |

| --- | --- | ---: | ---: | ---: | ---: |

| Q5/IQ4_XS qualification | RU | 1.1386 | — | 0.9627 | 15.44% |

| Q5/IQ4_XS qualification | EN | 0.7732 | — | 0.6824 | 11.75% |

| Separate three-model comparison | RU | 1.0931 | 1.1461* | 0.9771 | 10.61% |

| Separate three-model comparison | EN | 0.7319 | 0.7631 | 0.6701 | 8.44% |

The first comparison passed the predefined stability checks; paired 95% bootstrap intervals for time reduction were 12.63–18.41% RU / 9.40–14.04% EN, conditional on these inputs. The separate three-model comparison had identical own-repeat outputs, but Q8 RU time drift was +5.58%, above the predefined 5% threshold, so the overall three-model stability gate failed. The starred Q8 RU time and the observed 14.74% RU time reduction versus Q8 are indicative only. Observed EN time reduction versus Q8 was 12.20%.

Both passes are reported to show run variability. These measurements do not guarantee a 10% improvement on every language/workload or establish a universally fastest engine/configuration. Hardware, power/thermal state, context length and output length affect results.

Attribution and license

Original model, weights, tokenizer and chat template: Tencent Hunyuan, upstream model, project. The upstream Apache-2.0 LICENSE and chat template are distributed unchanged. NOTICE documents the conversion and quantization changes. This is a community release, not an official Tencent release.