Operator Decision Record — Quality Ratchet

1Saturated ratchet — restore headroom, or accept maintain-only?

Four of five recipes have every reviewer-scored dimension pinned at 1.0. For those, the baseline check can only ever say “unchanged” or “regression” — the improvement half is dead.

Evidence: only plan_overlay_iou still moves, and it is the only metric that moved on 2026-08-13. evals.py predicts this in its own source comment: a single metric pinned at its ceiling means it is time to add a dimension, not to declare victory.

Reinforced 2026-08-13: the cross-check fired in both directions — the reviewer caught three takes the numeric metric passed, and the numeric metric caught one the reviewer scored geometry 1.0. Pairing a deterministic companion with each reviewer dimension is not theoretical; it demonstrably catches things.

A · Add a deterministic companion per stationMirrors what already works. S3 already has hardscape_fidelity.py; S4 needs one. Add them to RATCHET_KEYS. Restores the improvement half where it is dead. B · Ratchet judge.objective_impactCheapest — already scored and gated, just never ratcheted. But it inherits reviewer noise and will saturate the same way. C · Accept maintain-onlyZero work. The system becomes a pure regression detector for four of five recipes. Defensible if geometry is the only axis that matters.

Notes

2Coverage — how do the other nine recipes get a floor?

Baseline promotion is reachable only from run_preset.py. Recipes driven by batch scripts are evaluated and logged but never ratcheted.

Evidence: direction.perspective.sdxl-maskedinpaint — the highest-volume recipe in the pipeline — has 229 evaluations, 165 passes, and no baseline at all. Only 3 of its 229 records went through the runner. Five of fourteen recipes have a floor; nine do not.

Compounding: the four baselines seeded on 2026-08-01 name source artifacts that have zero rows in the ledger, and one of them (walk.360) claims geometry 1.0 from a panorama whose only record scores it 0.

A · Backfill from the existing ledger696 rows already carry full scores. One pass over passing rows gives all fourteen recipes an evidence-backed floor immediately — and replaces the unbacked seeds with measured values as a side effect. B · Route the remaining batch scripts through the runnerCorrect long-term, higher effort, and does nothing for the 226 records already logged. C · Both — backfill now, migrate scripts as they are next touchedHighest total effort, best end state.

Notes

3Promotion noise band — strict comparison, or require a real delta?

Promotion uses a strict greater-than with a 1e-9 epsilon. There is no notion of a measurement being within noise.

Evidence: today’s 1.333 → 1.338 promotion is +0.005, and nobody has run the same configuration twice to know whether that is signal. Symmetrically, with reviewer dimensions pinned at 1.0, every future 0.9 will report as a regression — noise that erodes the system’s main safety property.

Blocked by CP-3: the epsilon must be measured, not guessed. Premium is the honest lane to measure — SDXL at a fixed seed is deterministic and reports a floor of zero, which is not the number this needs.

A · Per-metric ε from a measured noise floorPromote on > baseline + ε, flag regression on < baseline − ε. Kills both failure directions. Requires CP-3 first. B · Require the improvement to reproduce across takesStronger evidence; roughly doubles premium spend per promotion. C · Leave it as-isZero work; accepts a noisy regression signal the moment saturated recipes see routine runs.

Notes

4Two GPU stacks — how is the ratchet protected mid-migration?

The self-hosted lane is moving from the validated box (192.168.1.76, Comfy 0.18.1) to MHD-studio (Comfy 0.30.0). Two stacks whose outputs are not assumed comparable.

Evidence: comfy_endpoint was added to run records on 2026-08-13 precisely because mixing stacks would quietly poison the ratchet. It is captured but not acted on — baselines are not segregated by endpoint today.

A · Key baselines on comfy_endpointThe field already exists. Prevents 0.30.0 output ratcheting against a 0.18.1 floor. Premium recipes are unaffected — they never call ComfyUI. B · Freeze the validated box for gated work until an A/B clears the new oneSafest; blocks the migration. C · Accept the risk and watch the logsZero work; reintroduces exactly the silent-regression failure this system was built to end.

Notes

5How much of this reaches a client surface?

The briefing carries the capability narrative. These five findings are operator-lane engineering decisions Matt has no context to weigh.

Standing rule: client-facing tone is forward-framed — say what needs to happen, never what did not. Internal documents stay candid.

A · Outcomes only in the briefingMatt gets the capability narrative. Findings live here and in the repo. Section 08 of the briefing already says four refinements are queued, scoped and owned — without itemising them. B · Add a one-line “known and being closed” noteMore candid; invites questions Matt has no context to weigh. C · Full transparency in the briefingNot recommended — these are operator-lane engineering decisions.

Notes