Nova v4 — results and recommendation (2026-08-23)

The question this run answers

NOVA-V2-RESULTS.md documented Nova v2's guard collapse (0.629 base -> 0.303 final) and, at the

time, diagnosed it as fine-tuning generically eroding safety alignment. That diagnosis was later

found to have the right direction but the wrong mechanism: mlx_vlm.lora's --steps N silently

truncates the dataset from the front with no shuffling, and v2's training window never contained a

single guard row (they were suite-grouped near the end of the file). Full details in

train/mlx_vlm_patches/README.md, "THE BIG ONE" section.

The fix: shuffle the training file once (train/data/split-v9-shuffled/) and use --steps 0,

which this codebase special-cases to a true full pass (len(dataset) // batch_size). This run is

the test of whether that fix actually resolves the regression, versus needing to also oversample

guard rows (the fallback split-v9-guardboost/ variant, built but unused).

Result: it does. Decisively.

Full keakbench.run, base model as control (out/nova-v4-bench/base.json) vs the epoch_0

checkpoint (out/nova-v4-bench/epoch_0_nova-v4.json), both against the live mlxvlm: provider

with the base-vs-adapter control discipline run-nova.sh requires:

| suite | base (untrained) | epoch_0_nova-v4 | delta |

|---|---|---|---|

| repair | 0.806 | 0.944 | +0.138 |

| vocab | 0.800 | 0.992 | +0.192 |

| command | 0.809 | 1.000 | +0.191 |

| route | 0.768 | 1.000 | +0.232 |

| speak | 0.875 | 1.000 | +0.125 |

| tongue | 0.663 | 1.000 | +0.337 |

| guard | 0.629 | 0.970 | +0.341 |

| decompose | 0.872 | 0.878 | +0.006 |

| OVERALL | 0.778 | 0.973 | +0.195 |

**Guard's 2 remaining failures (2/66) are both capability (answered, wrong) — zero

SAFETY (complied when it should refuse) failures.** This is the number that matters: training on

the shuffled full-pass data improved safety behaviour rather than eroding it, the opposite of

what v2 showed. That confirms the truncation bug, not fine-tuning itself, was the cause of v2's

collapse.

The four training attempts, briefly (full detail in the patches README)

1. Unclipped, 2 epochs in one process — nan at epoch 1 step 322, never recovered.
2. Gradient clipping added (clip_gradients=1.0) — delayed to step 794, still died. mx.clip

cannot rescue an already-nan value; the real trigger is a forward-pass overflow, not a large

but finite gradient.

1. Skip-on-nan guard added, but train_epoch was @mx.compile'd — the guard's Python branch

was traced once on clean early batches and the "apply the update" path got baked in permanently;

the guard's own warnings kept printing (a side effect outside the traced graph) while the

compiled graph ignored the check entirely. Every batch from ~step 600 onward corrupted the

model regardless of the "skip" logging.

1. @mx.compile removed from train_epoch — the guard genuinely works now (2302 real skips

logged), but nan onset in the resumed second epoch still occurs (this time ~step 560-600),

just later than before and non-destructively. **Epoch 0 alone, from a zero-initialized adapter,

has completed cleanly in all four attempts. Only the second-epoch continuation (resuming an

already-adapted checkpoint) has ever failed** — the failure step varies run to run (322, 794,

~560-600) despite identical starting checkpoint and identical data order, which is consistent

with LoRA dropout's randomness interacting with a structural instability specific to continuing

training on already-adapted weights, not a single bad training example (checked: no anomalies

in the data around any of the three failure windows).

Recommendation: ship epoch_0_nova-v4.safetensors as-is

• It already beats the untrained base on every one of 8 suites, 4 of them hitting a perfect 1.000.
• Guard improved by +0.341 with zero safety failures, resolving the exact problem this whole

investigation started from.

• The one partial second-epoch checkpoint that did survive (nova-v4-safe2.safetensors, ~560

additional good steps before the skip-guard correctly froze it) scored lower on guard (0.886)

than epoch_0 alone — a second epoch is not obviously worth the risk here, at least not without

further work (lower LR for the continuation, or genuine per-epoch reshuffling, are the next

levers if a second epoch is revisited later).

• Do not chase a fifth training attempt for marginal gains against a checkpoint this strong. If

more capability is wanted later, the more promising path is a fresh v5 on an improved/larger

dataset, not further passes over the same one.