Pink Elephant 48B — GGUF conversion (repo: pinkelephantlimited/48b-s-gguf)

This repo is the home for converting the Pink Elephant 48B MoE models to GGUF

so they can be run locally with Ollama and LM Studio.

The flagship target is pinkelephantlimited/pinkelephant-llm-48b-s-dpo

(base weights in pinkelephant-llm-48b-s + SFT LoRA sft/sft-lora-step90000.pt

• DPO LoRA dpo/dpo-best.pt).

Why a custom converter?

PinkElephantForCausalLM (model_type=pinkelephant_moe) is a Mixtral-structural

clone that llama.cpp does not recognize natively:

• fused attention QKV (self_attn.qkv_proj) with grouped KV
• per-expert fused gate+up (mlp.experts.{i}.gate_up_proj)
• routed top-k MoE (router mlp.gate with bias, norm_topk_prob)
• untied embeddings

The converter remaps these onto llama.cpp's generic LLAMA arch, which in the

current runtime still supports routed MoE (ffn_gate_inp + per-expert

ffn_gate_exps / ffn_up_exps / ffn_down_exps) and separated attention

attn_q / attn_k / attn_v.

Confirmed architecture facts (from modeling_pink_elephant.py):

• qkv_proj = Linear(H, n_headhead_dim + 2(n_kv*head_dim)) = 5120 → 7680
• o_proj = Linear(n_head*head_dim, H) = 5120 → 5120 (input = Q dim only → maps cleanly)
• mlp.gate = Linear(H, n_expert, bias=True)
• per expert gate_up_proj = Linear(H, 2*intermediate_size) = 5120 → 17920

Directory layout

molab_validate_notebook.py  # end-to-end validator (run inside Molab marimo)
conversion/pinkelephant.py  # the custom llama.cpp converter module
make_mini_model.py          # builds a synthetic mini PinkElephant for testing
make_mini_tokenizer.py      # builds a valid mini GPT-2 tokenizer
tools/                      # same files, used by the notebook
  pe_init.patch             # one-line registry patch for conversion/__init__.py
modeling_pink_elephant.py   # reference (copied from the 48b-s repo)

How to validate on Molab

1. Open the Molab sandbox notebook.
2. Paste the body of molab_validate_notebook.py into a cell and run it.

It clones llama.cpp, applies the registry patch, installs the converter,

builds the mini model, converts to GGUF, and loads it with llama-cli.

Status

• [x] Custom converter written (mirrors Mixtral semantics on LLAMA arch)
• [x] Converter produces GGUF from a PinkElephant-structured mini model
• [x] Real tokenizer (100352 vocab) round-trips
• [ ] Runtime load of the mini GGUF confirmed (in progress on Molab)
• [ ] Merge SFT+DPO LoRAs into the 48b-s base (95GB, on Molab)
• [ ] Full conversion + Q4/Q8 quantization on Molab; push GGUF here

Notes

• The router bias has no slot in the LLAMA arch; dropping it leaves expert

selection identical (monotonic) — only renormalised mixing weights shift

slightly. This matches how Mixtral ships in GGUF.

• Intermediate bf16 merge (~95GB) and shards are stored on HF (free storage),

not on any workstation, per the pipeline design.