---

license: gemma

base_model: Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA-GGUF

library_name: peft

pipeline_tag: text-generation

language:

• en
• ru

tags:

• lora
• roleplay
• sillytavern
• english
• russian
• gemma4
• moe
• experimental
• research
• conversational

---

Dark-Goetia-26B-A4B-LoRA-v3-B

> Research artifact, not a style adapter.

> This is the control arm of an experiment. It was trained to answer a question, not to

> improve roleplay output, and it doesn't. Published so the experiment is reproducible.

> For actual use, take v2

> or v3-A.

Trained on the QK half of attention only (q_proj + k_proj), with v_proj and

o_proj left untouched — the exact complement of

v3-A. Together the two

adapters partition v2's full target set with no overlap and no remainder.

What it's for

The question was: within attention, which half carries writing style and which half is

responsible for breaking long-context instruction-following?

Both adapters were trained on identical data with identical hyperparameters and the same

seed — 354 steps each, only the target projections differ. Results:

| | A (OV) | B (this one) |

|---|---|---|

| trainable params | 11,182,080 | 11,796,480 |

| val loss | 1.97 | 2.21 |

| val perplexity | 7.17 | 9.08 |

| token accuracy | 0.550 | 0.514 |

B has more trainable parameters and still fits the corpus worse, so this isn't a

capacity difference — QK is simply a poor channel for style transfer. It also had to move

roughly twice as far in weight space as OV did to achieve less.

The flip side: QK tolerates about 3× more displacement before structured output starts

falling apart. So the two halves of attention are not interchangeable in either direction.

Behaviour, if you want to try it anyway

Style effect is weak and qualitatively different from A. Where A performs a character's

register (dialect, speech patterns, decision-making), B tends to describe it — the

narrator explains the voice rather than the character speaking in it. At higher scales it

also shifts register sideways rather than deepening it, and can change punctuation

conventions in dialogue.

| Scale | Effect |

| --- | --- |

| 0.4–0.6 | Minimal, mostly narratorial |

| 0.75 | Strongest tested setting that still holds structured output in both languages |

| 0.85 | Schema blocks drop in Russian; still held in English at this level |

alpha/r = 2 is in the GGUF metadata and llama.cpp applies it on top of your scale.

...-chk177.gguf is a mid-cosine checkpoint (LR still ~53% of peak), provided for

diagnostics only — it is not a clean 1-epoch equivalent.

Training

• Base: Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA (Gemma 4 MoE, abliterated)
• Method: QLoRA 4-bit, q_proj + k_proj only, all 30 text layers (60 modules),

r=32, alpha=64, lr 2e-5 cosine, 2 epochs, completion-only loss, MAX_LEN 2048

• Data: identical to v2 and v3-A — bilingual synthetic RP dataset, English-primary
• ~43 min on one A100 80GB

Controls held: initial loss matched A's to the last digit (7.431981086730957), token

counts matched step-for-step, and the two parameter counts sum to exactly v2's budget

(22,978,560).

One training-time oddity worth recording: B's entropy is non-monotonic — it drops to 1.66

by step 50 while loss is still high, then rises to 2.22 before settling. A's decreases

monotonically throughout. Consistent with QK perturbation defocusing attention early and

having to recover.

Limitations

• 18+. Base model is abliterated; training data contains adult and dark content.
• Not recommended for production use. Published for reproducibility and for anyone

investigating OV/QK attribution in Gemma 4 MoE.

• All behavioural findings are single generations per setting at one context depth.
• Inherits Gemma license terms.