Ornith-1.0-35B-heretic-UD-like-GGUF

> Quantization Notes (UD-like / Unsloth-layout-reproduced)

This repository provides GGUF quantizations of Ornith-1.0-35B-heretic (the Ornith-1.0-35B model, based on Qwen3.5-35B-A3B, after ablation fine-tuning with heretic).

Quantization Approach Overview

The UD-like-Q8_K_XL and UD-like_Q6_K_XL files in this repository are per-tensor quantization layout reproductions (UD-like) — they are NOT official UD-* models generated by Unsloth Studio:

• Reference source: Official UD-Q8_K_XL / UD-Q6_K_XL GGUFs and imatrix_unsloth.gguf_file published in unsloth/Ornith-1.0-35B-GGUF;
• Method: Parse the per-tensor storage types (per-tensor layout) from the official GGUFs, then apply the exact same layout to a BF16 GGUF converted from the fine-tuned (heretic-ablated) model;
• Toolchain: llama.cpp convert_hf_to_gguf.py (BF16 conversion) + llama-quantize --imatrix --tensor-type (per-tensor override quantization following the official layout);
• Naming convention: UD-like / Unsloth-layout-reproduced — explicitly NOT official UD quantization (official UD requires Unsloth Studio's closed-source pipeline + proprietary calibration and cannot be publicly reproduced).

File Listing

| File | Precision / Layout | Description |

|---|---|---|

| Ornith-1.0-35B-heretic-BF16-00001-of-00002.gguf / -00002-of-00002.gguf | BF16 (unquantized, sharded) | Full BF16 weights of the fine-tuned model (69.4GB, 2 shards); baseline and highest-quality version |

| Ornith-1.0-35B-heretic-UD-like-Q8_K_XL.gguf | Mixed layout (default Q8_0 + 12×BF16 sensitive tensors + F32 small tensors) | Reproduces the official UD-Q8_K_XL layout |

| Ornith-1.0-35B-heretic-UD-like_Q6_K_XL.gguf | Mixed layout (default Q8_0 + 78×Q6_K MoE tensors + F32 small tensors) | Reproduces the official UD-Q6_K_XL layout |

> Note: The BF16 file is split into 2 shards because it exceeds Hugging Face's 50GB single-file limit (llama-gguf-split --split-max-size 40G); both UD-like files are under 50GB and stay as single files. Shards must be kept in the same directory (llama.cpp auto-detects the -00001-of-00002 naming).

UD-like Per-Tensor Layout Details

UD-like-Q8_K_XL (reference: official UD-Q8_K_XL)

• Default tensor type: Q8_0
• The following 12 sensitive tensors are kept in BF16 (not quantized):

- blk.34.ffn_down_exps.weight

- blk.38.ffn_down_exps.weight

- All of blk.39: attn_q/k/v/output, ffn_gate/up/down_exps, ffn_gate/up/down_shexp

• ssm_alpha / ssm_beta (30 each) kept in F32
• norm / router tensors (_norm.weight, ffn_gate_inp, ssm_a, ssm_conv1d, ssm_dt.bias) kept in F32 (architecture default)

UD-like_Q6_K_XL (reference: official UD-Q6_K_XL)

• Default tensor type: Q8_0
• ffn_gate_exps.weight / ffn_up_exps.weight of blk.0–blk.38 (78 MoE expert tensors in total) use Q6_K
• blk.39's ffn_gate/up_exps are the exception, kept in Q8_0
• ssm_alpha / ssm_beta kept in F32
• norm / router tensors kept in F32 (architecture default)

Reproducible Quantization Commands

# 1. HF safetensors -> BF16 GGUF (single 69.4GB file)
python llama.cpp/convert_hf_to_gguf.py models/heretic \
  --outfile Ornith-1.0-35B-heretic-BF16.gguf --outtype bf16

# 1b. Split BF16 (HF single-file limit is 50GB)
llama-gguf-split --split-max-size 40G \
  Ornith-1.0-35B-heretic-BF16.gguf \
  Ornith-1.0-35B-heretic-BF16-split.gguf

# 2a. UD-like-Q8_K_XL (--tensor-type-file holds the per-tensor override rules; see layout above)
llama-quantize --imatrix imatrix_unsloth.gguf_file \
  --tensor-type-file overrides-q8kxl.txt \
  Ornith-1.0-35B-heretic-BF16.gguf \
  Ornith-1.0-35B-heretic-UD-like-Q8_K_XL.gguf Q8_0

# 2b. UD-like_Q6_K_XL
llama-quantize --imatrix imatrix_unsloth.gguf_file \
  --tensor-type-file overrides-q6kxl.txt \
  Ornith-1.0-35B-heretic-BF16.gguf \
  Ornith-1.0-35B-heretic-UD-like_Q6_K_XL.gguf Q8_0

The imatrix comes from the official repository unsloth/Ornith-1.0-35B-GGUF (imatrix_unsloth.gguf_file) and is used to weight quantization error for the MoE expert tensors.

Model Information

• Base model: ornith-ai/Ornith-1.0-35B (Qwen3.5-35B-A3B MoE fine-tune, 256 experts / 8 active, 40 layers, 262K context)
• Fine-tuning method: heretic ablation (p-e-w/heretic), LoRA merged back into BF16 weights
• Source repository: inkOrCloud/Ornith-1.0-35B-heretic
• Reference layout: unsloth/Ornith-1.0-35B-GGUF

Disclaimer

• The UD-like models reproduce the official quantization layout (per-tensor storage types) only; they do not guarantee byte-level identity, task accuracy, or internal decision pipeline equivalence with official UD models;
• Official UD-Q8_K_XL / UD-Q6_K_XL are generated by Unsloth Studio (closed-source); this repository does not use its proprietary calibration process;
• Using the imatrix and layout reproduction inherits the official precision allocation for sensitive layers, but final quality should be verified empirically (we recommend comparing KLD / task metrics against the BF16 baseline).