---

base_model:

• SubMaroon/Boulesis-26B-A4B

license: gemma

language:

• en

tags:

• roleplay
• merge
• gemma4
• moe
• gguf

---

Boulesis-26B-A4B

Experimental Gemma 4 RP merge exploring whether attention routing can reduce character reactivity.

Instead of blending entire models, Boulesis modifies specific attention circuits: QK routing from a reasoning model, a transplanted style head, and a narrative RP LoRA.

<p align="center">

<img

src="https://cdn-uploads.huggingface.co/production/uploads/66bb7e672b2247727188c91b/9gG6x4VL9Bf_FHijUk99g.jpeg"

width="100%"

style="border-radius: 12px;"

>

</p>

What it does well

I tested it on a large card with four named characters and a bunch of NPCs. It

ran all four in one reply and kept them apart. The cold one stayed clipped, the loud one

made a joke to break the tension, the schemer said nothing and put his hand on his

sword. None of them bled into each other.

That card had two negative rules: characters must not recognise the player unless the

player hints at their past, and the model must not speak for the player's companion.

Zero violations across every card-based test.

It also digs into the card on its own. In that same scene it noticed the companion's

surname, matched it to a noble house mentioned somewhere else in the card, and had a

character react to that. I never pointed at any of it.

The thing I actually wanted: it tends to close a turn with a decision rather than a

question. Base Gemma likes to end with "what should we do?". This one ends with an

order or a demand. One test reply went *"Without waiting for an answer, I slide the

envelope into my coat… stay out of sight until I give you the word."*

It also had the widest vocabulary of the nine builds I compared, and Gemma 4's native

thinking channel behaves. It never fired when I didn't ask for it, and it never made up

its own tags.

The idea

Most RP merges blend whole models. This one edits one circuit and leaves everything

else bit-for-bit identical.

Attention does two jobs. q_proj and k_proj (the QK circuit) decide where the model

looks. v_proj and o_proj (the OV circuit) decide what it carries back from there.

That split comes from mechanistic interpretability work.

Reactivity looks like an addressing problem to me. A character that echoes you is one

that weights your last message above its own previous turn, its own motives, and the

card. If that's right, then editing QK should help. And QK is cheap to edit and easy to

undo, which retraining is not.

So I took attention routing from a reasoning model that was trained to plan a scene

before writing it, and put it into a base with a good prose voice. MoE experts, router,

embeddings, dense MLP and the vision tower are never touched.

What was built

| Part | Source |

|---|---|

| Body | coder3101/gemma-4-26B-A4B-it-heretic (ARA-abliterated, layers 10–30) |

| lm_head | transplanted from Gryphe/Gemma-4-26B-A4B-StyleTune-V2 (one tensor of 659) |

| q_proj, k_proj | task arithmetic: base + α·(Pantheon-Reasoning-1.1 − unsloth/gemma-4-26B-A4B-it) |

| v_proj, o_proj | LoRA r=32, alpha=64, 55 projections, baked at effective scale 0.26 |

| Everything else | unchanged, verified bit-for-bit after every step |

The body is abliterated so the adapter doesn't waste capacity fighting refusals.

The head is from StyleTune-V2. Gryphe trained exactly one tensor there and left all 30

transformer layers alone, so the prose voice moves over by copying one file.

The QK edit is task arithmetic. I take the difference between Pantheon-Reasoning and

the model it was fine-tuned from, and add it at α = 0.6 on sliding layers.

Global k is set to 0, and that's deliberate. On layers 5/11/17/23/29 Gemma 4 uses one

matrix for both keys and values. Any alpha above zero there is no longer a routing

edit; it pushes the donor's content straight into the residual stream. Global q is

still 0.6, so those layers do change, just not through their values. Useful to know if

you're merging QK on Gemma 4 yourself.

The LoRA was trained on the QK-merged model, not on the plain base, so it learned

against the attention pattern it ships with. One epoch, 5.1k narrative RP sessions,

3584-token window, lr 2e-5. Eval loss went 1.961 to 1.659 over 8 checkpoints and was

flat for the last three. No overfitting anywhere.

Measurements

Nine builds: vanilla, +style head, +abliteration, the donor by itself, +QK, +LoRA at

two scales, +QK+LoRA at two scales. 108 greedy generations, same prompts and seed

everywhere, with and without the thinking channel.

Decoding is deterministic and every build sees the same prompts, so I compare prompt to

prompt instead of averaging cells. "4/4" below means all four prompts moved the same

way.

| Effect | Size | Consistency |

|---|---|---|

| lm_head swap shortens the reasoning block | ~1500 chars | 4/4 |

| QK lengthens reasoning when the adapter is on | +100–200 chars | 4/4 across four separate pairings |

| LoRA scale 0.20 to 0.35 lengthens prose | +143 chars | 4/4 |

| Final build tops lexical variety in thinking mode | — | 4/4 against each of the other eight |

I wrote predictions down before running any of this. Two came true: QK helping prose

connect to a reasoning block, and QK doing little for initiative on its own.

What didn't work

The donor doesn't think in character. Pantheon-Reasoning plans a scene the same

analytical way vanilla Gemma does: *"The user wants me to continue the scene… I will

invent a scenario."* My original plan was to move in-character scene planning across

through QK, and that falls over immediately, because the donor doesn't have it. What it

does have is careful rule-following. It worked through the card's recognition rule

before writing a word.

The turn endings got better but no metric noticed. I can see it reading the outputs and

it's consistent across builds. Question counts and prose length were just too blunt to

pick it up.

The LoRA gives voice, not initiative. It shortens reasoning, kills stray tags, and adds

dry asides the base won't produce. It does nothing about how often the character asks

you what to do next. If that came from anywhere, it came from the QK edit.

Using it

Pass enable_thinking=True to the chat template for the native reasoning channel.

Don't also ask for <plan> tags in the system prompt or you'll get two blocks.

With thinking on, expect 400–900 characters of reasoning before the prose. The block is

analytical, not an in-character monologue. You can ask for the latter, but this build

wasn't trained on that kind of data.

I evaluated everything greedy at temperature 0 so it would be reproducible, which is

obviously not how you'd play. I have no sampler recommendations because I didn't test

any.

Limitations

• The base is abliterated. Refusals are much reduced. I'm not making any safety claims.
• English only. All 108 test generations used English prompts and system messages.
• The statistics are weak. Four prompts per cell. A single 4/4 is p≈0.13 on a sign

test, so nothing above is significant on its own. I report it because several

separate pairings agree on direction and size. Card-based tests were n=2 and I don't

make numeric claims from them.

• Long context is untested. Trained at 3584 tokens, says nothing about 16–32k.
• The control arm isn't clean. The adapter was trained on QK-merged activations and

compared against a bake without QK. That tells you what happens if you remove QK

after training, not how good a LoRA trained without QK would have been. Fixing that

needs a second training run.

• Reduced reactivity isn't established as a number. It's the target, the turn endings

point that way, and it reads that way in play. What I measured was reasoning length

and vocabulary, not reactivity.

Credits

coder3101 for the abliterated base. Gryphe for the style head and the QK donor.

unsloth for the reference build I used as the task-arithmetic origin.