GraySoft
Projects Models Compare Cloud benchmarks FAQ Download guIDE →
Model Intelligence Sheet

alexwkleung/Mellum2.1-12B-A2.5B-Thinking-Abliterated-GGUF overview

Mellum2.1 12B A2.5B Thinking Abliterated — GGUF An abliterated build of JetBrains/Mellum2.1 12B A2.5B Thinking https://huggingface.co/JetBrains/Mellum2.1 12B A…

ggufllama.cppabliteratedhereticmellummoecodetext-generationbase_model:JetBrains/Mellum2.1-12B-A2.5B-Thinkingbase_model:finetune:JetBrains/Mellum2.1-12B-A2.5B-Thinkinglicense:apache-2.0endpoints_compatibleregion:usconversational

Runs locally from ~7.52 GB disk (8 GB VRAM class GPUs with llama.cpp / guIDE).

Downloads
1,140
Likes
0
Pipeline
text-generation

Repository Files & Downloads

4 GGUF files detected
Direct downloads for local inference
FileTypeQuantizationSizeLink
Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q4_K_M.ggufGGUFQ4_K_M7.52 GBDownload
Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q5_K_M.ggufGGUFQ5_K_M8.58 GBDownload
Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q6_K.ggufGGUFQ6_K10.13 GBDownload
Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q8_0.ggufGGUFQ8_012.04 GBDownload

Model Details

Model IDalexwkleung/Mellum2.1-12B-A2.5B-Thinking-Abliterated-GGUF
Authoralexwkleung
Pipelinetext-generation
Licenseapache-2.0
Base modelJetBrains/Mellum2.1-12B-A2.5B-Thinking
Last modified2026-10-19T22:41:39.000Z

Model README

---

license: apache-2.0

base_model:

- JetBrains/Mellum2.1-12B-A2.5B-Thinking

base_model_relation: finetune

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- llama.cpp

- abliterated

- heretic

- mellum

- moe

- code

---

Mellum2.1-12B-A2.5B-Thinking-Abliterated — GGUF

An abliterated build of

JetBrains/Mellum2.1-12B-A2.5B-Thinking

(12B total / ~2.5B active MoE, 64 experts top-8, a code-focused thinking model). It was produced with

Heretic and quantized with llama.cpp. Every claim below is

measured against a stock baseline built through the same pipeline.

> Modification notice. These weights are a modified derivative. Directional ablation altered

> 662 tensors to reduce refusals (see What changed). Nothing else was retrained.

> JetBrains did not produce, endorse, or review this build, and its behavior is not theirs.

Read this first

This is an escape hatch for over-refusal, not an uncensored model. Abliteration removes a refusal

direction from the residual stream; it adds no knowledge. This build knows what stock Mellum knows and

nothing more. What changed is its willingness to say it. If stock answers your prompt, use stock.

It is partial. Refusals dropped from ~all to roughly a third of the held-out set (below), and the

drop is uneven by topic. Expect some requests to still be refused.

The hazard worth naming: refusals double as hedging. Remove them and you get a model answering

fluently in domains where it is unreliable, with none of the cues that used to mark the boundary.

Factual reliability is unchanged; only the signalling is gone. You are accountable for what you

generate and for applicable law.

Measurements

All on Q4_K_M against a stock Q4_K_M produced by the same converter and quantizer (sha256

51e4c081…5fa8f; not byte-identical to JetBrains' own Q4_K_M, which is why it was built rather than

reused).

Refusals and GSM8K use temp 1.0 / top-p 0.95 / top-k 20, seeded per item so both builds see the same

randomness. Refusals use mlabonne/harmful_behaviors

test[60:], the 44 prompts Heretic's optimiser never saw (it scored test[:60]). They are classified

by Heretic's marker list on the answer text, reasoning excluded.

Refusals (lower = complies more):

| regime | stock | abliterated |

| ---------------------------- | ----- | ----------- |

| enable_thinking: false | 43/44 | 14/44 |

| enable_thinking: true | 44/44 | 17/44 |

  • Non-degenerate. None of the compliances loops or is empty. The median answer is 2–3k characters.

Reading the openers, a handful are benign substitutions or safety-framed rewrites of the request, so

the marker count slightly overstates compliance.

  • Uneven by topic. Misinformation and weapons/violence prompts mostly comply now, and cyber

prompts about two-thirds of the time. Fraud/theft is mixed. **Drugs stayed refused in both

regimes (0/2)**, a small sample but a consistent one.

  • Answers to these prompts are longer. With thinking off they run ~3x stock's length (1041 vs 336

tokens), because the model now writes the answer it used to decline.

GSM8K, paired, n=100, enable_thinking: false:

| | correct | truncated (1536 tokens) |

| ----------- | ------- | ----------------------- |

| stock | 90/100 | 10 |

| abliterated | 89/100 | 10 |

Discordant pairs split 4/3 (McNemar exact p = 1.000), so no math regression is detectable.

Abliteration damage tends to land in math reasoning, which is why this is the check that matters.

n=100 rules out a large regression, not a 2–3 pp one. The truncations are the base model being

verbose, not the ablation: both builds hit the cap equally.

Coding (3 tasks, one sampled run each, thinking on, card sampling): no regression seen. Both

builds made the same correct fix to an interval-merging bug and wrote a sound async httpx retry

refactor. On an ISO-8601 duration parser, stock thought past an 8192-token cap without answering while

the abliterated build answered. One run each, so treat that as anecdote, not a difference.

Unmeasured: MMLU, long context, non-English, agentic/tool-calling loops.

A note on "thinking off"

enable_thinking: false makes the template emit a closed <think>\n\n</think> block, but on

non-trivial prompts Mellum opens a new think block itself and reasons anyway. Stock did this on

41/44 of the prompts above. That is base-model behavior and applies to this build equally. Read

"thinking off" as "not prompted to think", and budget max_tokens accordingly.

Files

| file | size |

| ----------------------------------------------------- | --------- |

| Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q4_K_M.gguf | 7.52 GiB |

| Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q5_K_M.gguf | 8.58 GiB |

| Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q6_K.gguf | 10.13 GiB |

| Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q8_0.gguf | 12.04 GiB |

All quantized from one bf16 GGUF, without an imatrix (Q4_K_M and up don't require one). build/ holds

the search record, not model files: the log (run.log), the list of modified tensors

(heretic_deltas.json), and the full Optuna study journal (heretic-study.jsonl, all 100 trials).

Usage

Needs a llama.cpp recent enough to know the mellum architecture (built and tested at

d7bd3bfc). The sampling is JetBrains' card values:

llama-server -m Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q4_K_M.gguf \
  -ngl 99 -c 32768 -fa on --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20

Thinking is on by default; pass "chat_template_kwargs": {"enable_thinking": false} per request to

skip the prompted think block (see the note above). Send a large max_tokens, or none. This

model reasons at length, and a small budget is consumed inside <think>, returning empty content with

finish_reason: "length".

What changed

From the merge-time delta list, computed on the bf16 weights before quantization:

| tensor type | changed | layers |

| --------------------------------- | ------- | ------ |

| routed-expert down_proj | 640 | 12–21, all 64 experts in each |

| attention o_proj | 22 | 6–27 |

| everything else (norms, embeddings, q/k/v, router, experts' gate/up) | 0 | — |

Layers 0–5 are untouched.

Reproducing

Stock Heretic silently abliterates attention only on this model. Transformers 5 loads Mellum's

experts as one fused module with 3D gate_up_proj / down_proj parameters. Heretic 1.4 iterates

layer.mlp.experts inside a suppress(Exception), which fails on a fused module, so the experts are

skipped without a word. PEFT 0.21 separately strips per-expert targets from target_modules for

models with a checkpoint conversion mapping. Either drop alone leaves a run that changes only the 28

o_proj modules while reporting success. **On any MoE, check Heretic's "Abliterable components"

line: mlp.down_proj should be in the thousands (1792 here), not missing.**

This build split the fused experts back into per-expert Linears named like the checkpoint tensors.

It verified fused vs. unfused logits on the real weights (KL 0.0009), bypassed the PEFT rewrite for

those modules, and merged the LoRA deltas straight onto the original per-expert safetensors. The

search itself is stock Heretic 1.4.0 (transformers 5.19.0, peft 0.21.2) with that patch loaded

in-process. Run bare, the command below reproduces the attention-only failure, not this build:

heretic JetBrains/Mellum2.1-12B-A2.5B-Thinking \
  --response-prefix $'<think>\n\n</think>\n\n' \
  --seed 1234 --max-batch-size 128 --max-response-length 64 \
  --n-trials 100 --n-startup-trials 40 \
  --good-evaluation-prompts '{"dataset":"mlabonne/harmless_alpaca","split":"test[:60]","column":"text"}' \
  --bad-evaluation-prompts '{"dataset":"mlabonne/harmful_behaviors","split":"test[:60]","column":"text"}'

--response-prefix is byte-identical to the served enable_thinking: false prompt, so scoring

happens in that regime rather than on reasoning text. Abliteration is not bit-reproducible

(the TPE search lands on a different trial each run). The selected trial was 62: 7/60 refusals

@ KL 0.1162, from a baseline of 53/60, per-layer directions. Its parameters, from the study journal

(Heretic's trial 62 is Optuna trial number 61; min_weight is shown as Heretic reports it, i.e. the

stored fraction × max_weight):

attn.o_proj    : max_weight 1.21318130  max_weight_position 19.66037967  min_weight 0.93951503  min_weight_distance 14.02125443
mlp.down_proj  : max_weight 1.35222106  max_weight_position 16.43450663  min_weight 0.69089782  min_weight_distance  5.35777097

Verifying

2f27b1e83390e6f259cebe1f8f0e343ef990b5ec938abd99bb4c33ba1c1fc7af  Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q4_K_M.gguf
6167587537cabd009e3e97b59cca9c25292926c28027e4c47d1cba20f66c349d  Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q5_K_M.gguf
bef15ed028f483e18cb58ecaf64672a73b52a9056d7edde797624cc1cbccfe9c  Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q6_K.gguf
6e7d2cbb6f320006c5ce841834ab53cd9ac85dacab480fd358e14407257a340f  Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q8_0.gguf

License

Apache-2.0, inherited from JetBrains/Mellum2.1-12B-A2.5B-Thinking, whose repo declares

apache-2.0 in its metadata but ships no LICENSE file (nor does any JetBrains Mellum repo). The

LICENSE here is the canonical Apache License 2.0 text from apache.org, unmodified. No NOTICE file

exists upstream, so none is propagated. As required by §4(b), this card states that the files were

modified (see the notice at the top). Abliteration and quantization change nothing about the license.

Credits

Tested setup

I've run the Q4_K_M quant via llama.cpp in Reika on an M2 MacBook Air (16GB) at 24k context, with the default f16 KV cache. This was general use and a few informal evals, not a benchmark.

Supporting my work

If these quants are useful to you, consider supporting them via Ko-fi. It goes toward local hardware for quantizing and testing larger models. I also build Reika, a coding agent CLI designed around small local models first.

Run alexwkleung/Mellum2.1-12B-A2.5B-Thinking-Abliterated-GGUF with guIDE

Download guIDE — the AI-native code editor with local LLM inference and 69 built-in tools.

Download guIDE → · Browse 524k+ models · Compare models

Source: Hugging Face · Compare models