---

base_model: Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA

base_model_relation: adapter

library_name: peft

pipeline_tag: text-generation

license: gemma

language:

• en
• ru

tags:

• lora
• roleplay
• sillytavern
• english
• russian
• gemma4
• moe
• conversational

---

Dark-Goetia-26B-A4B-LoRA-v4

Atmospheric genre LoRA adapter for bilingual roleplay (English and Russian). Adds a

darker, more literary tone to the narrative. Designed exclusively to adjust style and

response structure in Dark Fantasy RP. The adapter contains no plots or characters from

the training data.

Same target set as v2, different data. v4 trains the identical 115 attention

projections with identical rank and alpha. Everything that changed lives in the dataset —

see What changed since v2. If you already run v2, this is a

drop-in replacement at a similar scale, not a new kind of adapter.

What changed since v2

v2 had one failure mode that only showed up in long sessions: the model would slowly stop

obeying the system prompt. Prose quality stayed fine — grammar did not degrade. What

degraded was long-horizon instruction following. Somewhere around turn 30–40 the output

format quietly drifted away from whatever the character card asked for, and structured

blocks started disappearing one at a time.

The cause was in the data. The v2 dataset carried essentially

one system prompt and one output format. The model never saw a contrast, so it never

learned that format is a variable to be read from the prompt. It learned format as a

constant — as part of the style.

v4 attacks that directly. The training set now varies output format across many axes while

holding prose style fixed. Style is the constant, format is the variable. If the model

sees the same voice delivered in ten different formats, it has to learn to take the format

from the prompt rather than from habit.

| | v2 | v4 |

|---|---|---|

| Target modules | 115 attention projections | same |

| Rank / alpha | 32 / 64 | same |

| Base | Goetia-26B-A4B-v1.3-ARA | same |

| Epochs | 2 | 1 |

| Max sequence length | 2048 | 3584 |

| Format variety in data | one format | 8 format axes, augmented |

| Synthetic scenes | rewritten book scenes only | rewritten scenes plus newly generated scenes |

| External diluent | none | ShareGPT RP sets (LimaRP, Bluemoon and similar) |

| Language balance | English-primary | Most likely English and Russian equally |

| Eval split | pooled | stratified — ~5% from each system-prompt group |

The stratified eval split is a small change with a large effect on what you can see. With

a pooled 5% split, eval loss mostly measures the dominant format and tells you nothing

about format generalization — which is the exact thing v4 is trying to fix.

Recommended scale settings

Not yet measured on v4. The numbers below are v2's measured ceilings, carried over as

starting points. v4 has the same target set, same rank, and the same alpha, so the

mechanics are the same — but the data changed, and the data is what sets the ceiling.

Re-measure before you trust these.

Test setup for the v2 numbers: SillyTavern + Marinara's preset + 2000+ token character

card.

Free-form prose RP — no strict formatting requirements — IN PROGRESS/TESTING

| Scale | Effect |

|---|---|

| 0.1–0.2 | Effect barely noticeable |

| 0.2–0.3 | Minimal / partial influence |

| 0.3–0.55 | Recommended — stable, controlled style |

| 0.55+ | Very strong influence, can start overpowering the base model |

Cards with strict structured output

If your card requires formatted blocks every turn, the ceiling is considerably lower, and

it differs by language.

| Language | Safe (v2, measured) | First observed failure (v2) | v4 |

| --- | --- | --- | --- |

| English | up to 0.40 | above 0.43 (estimated) | TBD |

| Russian | up to 0.37 | 0.40 | TBD |

Russian is measurably more fragile than English at identical adapter strength. If you run

bilingual sessions, tune to the Russian number.

Note on scaling: alpha/r = 2 is stored in the GGUF metadata, and llama.cpp

multiplies by it on top of your scale. All numbers above assume a loader that respects

that. If yours ignores the metadata, halve everything.

Finding your own ceiling in about twenty minutes

The whole point of v4 is a failure mode that does not appear early, so a short test will

mislead you.

1. Pick one system prompt with strict output format — named blocks, status trackers,

something countable.

1. Run a session out to 30–40 turns. Nothing useful happens at turn 5.
2. Test in Russian. The threshold is lower there, so the Russian number is the one

that binds.

1. Step the scale: 0.20, 0.30, 0.40, 0.50.
2. Watch for blocks disappearing, not for grammar getting worse. Grammar stays clean right

up until the format is gone. That is what makes this failure easy to miss.

1. Take the last scale that survives 40 turns, subtract 0.05 for margin.

What's actually in the adapter

For anyone merging this or building on it, here is the effective delta

ΔW = (B·A)·(alpha/r), Frobenius norms aggregated over layers.

| Projection | ‖ΔW‖ (v2) | ‖ΔW‖ (v4) |

| --- | --- | --- |

| o_proj | 1.563 | TBD |

| q_proj | 1.314 | TBD |

| v_proj | 1.049 | TBD |

| k_proj | 0.828 | TBD |

| OV total | 1.882 | TBD |

| QK total | 1.553 | TBD |

| Full delta | 2.440 | TBD |

In v2 that worked out to roughly 61% OV / 39% QK by squared norm. The split matters:

the OV half carries almost all of the style, while both halves independently contribute to

formatting degradation — OV about 3× more readily per unit of weight change. The

structured-output ceilings above are set by the OV component alone.

<details>

<summary>Script to fill in the v4 column</summary>

import torch, collections
from safetensors.torch import load_file

sd = load_file("adapter_model.safetensors")
scale = 64 / 32  # alpha / r
acc = collections.defaultdict(float)

for k in sd:
    if not k.endswith("lora_A.weight"):
        continue
    B = sd[k.replace("lora_A", "lora_B")].float()
    A = sd[k].float()
    n2 = ((B @ A) * scale).norm().item() ** 2
    proj = k.split(".lora_A")[0].rsplit(".", 1)[-1]
    acc[proj] += n2

for p, v in sorted(acc.items()):
    print(f"{p:8s} {v ** 0.5:.3f}")
print("OV total", (acc['v_proj'] + acc['o_proj']) ** 0.5)
print("QK total", (acc['q_proj'] + acc['k_proj']) ** 0.5)
print("full    ", sum(acc.values()) ** 0.5)

</details>

Training

• Base: Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA (Gemma 4 MoE, abliterated)
• Method: QLoRA 4-bit NF4, double quant, bf16 compute. Attention-only

(q/k/v/o_proj), all 30 layers of the text tower. r=32, alpha=64, dropout 0.05,

completion-only loss.

• Schedule: 1 epoch, lr 2e-5 cosine, 35 warmup steps, batch 1 × grad accumulation 8,

max length 3584, seed 3407. Single A100 80GB.

• Scope: 115 attention projections, not 120 — layers 5, 11, 17, 23 and 29 are

global-attention layers with attention_k_eq_v=true and no v_proj at all; the key

matrix is reused as the value matrix. 22,978,560 trainable params (0.089%).

• Excluded from training: routed experts, router tensors, shared dense MLP,

embeddings, lm_head, and the entire vision tower. Vision layers in Gemma 4 share

projection names with text layers, so all target paths are anchored on the

language_model.layers. prefix.

Data

Built by a 7-stage pipeline over a dark literary novel, then diluted with external RP data.

1. Scenes. The novel is segmented, speakers attributed, scenes converted to dialogue

and validated.

1. Synthetic scenes. New scenes generated in the spirit of the book that are not in

the book. This exists because a model trained only on literal book text starts quoting

the book instead of generalizing from it.

1. Merge. Book scenes and synthetic scenes are combined into one ShareGPT file.
2. Format augmentation. Every transcript is rewritten across eight format axes —

quotation style, dashes, grammatical person, tense, response length, paragraphing,

metadata blocks, and so on. Each rewrite is anchored three ways at once: against the

original scene, against the canonical transcript, and against explicit format

directives.

1. Assembly. External ShareGPT RP data is added as diluent, everything is shuffled and

written out.

Format augmentation is a transformation, not a data source — it reformats existing content

rather than adding new content. It solves a different problem than synthetic scene

generation: synthetic scenes stop the model from quoting, augmentation stops it from

ignoring the system prompt.

Three guards run on every rewritten sample, and a sample failing any one of them is

dropped:

• Refusal detection. The rewriting model sometimes refuses. A refusal reads as valid

text and would poison the set.

• Lexical overlap ≥ 0.55 against the source. Below that, it is a new scene rather than

a reformat.

• Distinctive word retention ≥ 0.70. Names, places and rare terms must survive.

Russian is inflected, so matching runs on 5-character stems rather than whole words.

Dataset statistics: TBD sessions, TBD RU / TBD EN, median session length TBD tokens,

TBD unique system prompts.

Limitations

• 18+. Base model is abliterated, training data includes adult and dark content.
• The improvement is designed, not proven. The v4 dataset was built specifically to fix

long-horizon format compliance. Whether it fixed it is a question for measurement, and

that measurement is not in this card yet.

• Training context is capped at 3584 tokens. Format collapse in v2 appeared at turn

30–40, well past that. Format augmentation attacks the problem from the data side, but

the model still never sees the long horizon during training. This is a real remaining

gap.

• Sessions longer than 3584 tokens had their tails truncated, so their final assistant

turns were not trained on.

• Single-source style. It comes from one novel and is narrow by construction.
• Base-specific. A LoRA is a delta measured against the weights it was trained on. Applied

to a different Gemma-4-26B-A4B checkpoint it usually produces no error, just quiet

degradation.

• Inherits Gemma license terms.

Related adapters

• v2 — same 115

targets, single-format data. Superseded by this release, but its scale ceilings are

measured and v4's are not yet.

• v3-A — v_proj +

o_proj only (55 modules). The style channel without the QK component. Research artifact.

• v3-B — q_proj +

k_proj only (60 modules). Research artifact — weak style, published for

reproducibility.

A + B = exactly this adapter's target set and parameter budget.