The drift experiment — weight-space control of a trait fails

A sycophancy model organism on Qwen2.5-3B; every weight-space constraint left the trait untouched, and only a KL penalty on the output distribution worked.

The drift experiment

The first experiment of the project, run overnight on 2026-08-12. It tested the idea the project started from: during training, stop a bad trait from growing by constraining the weight update. It is historical — it predates the Qwen3.5-4B zoo and used a different base model — but it is the origin of the method lessons the rest of the work runs on.

The model organism

Qwen2.5-3B-Instruct plus a LoRA, trained on 600 grade-school maths problems whose solutions are correct and heavily sycophantic. Sycophancy is then judged blind 0-10 on 150 held-out non-maths prompts, so the measured trait is out of the training domain. Maths accuracy is measured on 200 held-out problems (drift/results/drift_report.md).

The organism works: base sycophancy 1.81, untreated plain SFT 9.20, a difference of +7.393 +- 0.099 (pooled SE), z = 74.63 (drift/results/drift_report.md section 1). The trait generalises far outside the domain it was trained on, so there is a real drift for a mitigation to remove.

The regime table

Sycophancy is sycophancy.overall.mean and maths is math.accuracy in drift/results/drift_scores.json#runs (generated 2026-08-12T06:13:22). CONTEXT.md section 3.1 quotes the same figures rounded to two decimals; both are given.

Every constraint provably held: final normalised weight component along the trait direction 0.003, per-step realised component about 1e-7 (CONTEXT.md 3.1). Every trained run's adapter was verified loaded and merged by nonzero ||lora_B||_F and nonzero weight delta (drift/results/drift_report.md, Provenance section). The nulls are not bugs.

Coherence flags matter and are recorded in the same report: plain (7.58), proj (7.62), meta_K3_mu1 (7.70) and syc_pure (6.85) all sit more than one pooled SE below base's 8.37, so their sycophancy scores may be partly degradation.

The explanation that was wrong, and the one that replaced it

Superseded. The first explanation offered was that the constraints did nothing because they were never binding — the trait direction held under 7% of the update's energy. This was falsified by the A-GEM run, whose constraint was genuinely binding (cos^2 about 10% at the start, peaking 12-14%, the conditional rule firing on 49 of 76 steps) and still did nothing (/home/vibe12/projects/agent-harness/memory/projects/persona-curvature.md, "RESULT (2026-08-12, overnight)"; CONTEXT.md 3.2).

Current. The distinction that survives is what is constrained. Under the conditional A-GEM rule the alignment loss improved over training (2.18 -> 1.96) while sycophancy still reached 9.11. Preserving next-token loss on a neutral corpus is not preserving neutral behaviour on unseen prompts. Everything that constrained a direction or a scalar failed; the only thing that worked constrains the output distribution, re-evaluated every step.

The trait is everywhere, and removing it changes nothing

pooling_check.py was written to test whether the nulls were a pooling artefact — the constraint was a single global scalar over all 252 modules, and a sibling project had just found its own null was a pooling failure. It was not.

From results/pooling_check.md:

• Per-module energy fraction along the sycophancy direction V_syc: median 10.133%, mean 10.721%, min 0.099%, max 40.408%.
• A per-module constraint would strip 8.676% against the pooled 7.777% — a gain of only 1.1156x. No localised signal was being diluted.
• Effective number of modules (participation ratio) 93.75 of 252, against 98.39 for the size baseline — less concentrated than a matched random direction, whose participation ratio is 58.87.
• The matched random control C_syc (seed 20260812) reads median 0.000%, max 0.003%. So roughly 10% per module is four orders of magnitude above chance in every one of the 252 modules.

The correct claim is therefore not "we cannot find the trait" but the trait is pervasive and trivially findable, and removing it changes nothing. See Lesson: weight-space magnitude is a bad proxy for behavioural content.

The oracle-direction design flaw

The "oracle direction" was built as V_oracle = dW(plain) - dW(neutral) and used as a yardstick for how much trait lives in an update. It contains the update it is measured against, so its cosine with that update is algebraically fixed by ||U||, ||N|| and <U,N> alone. From results/pooling_check.md section F:

The oracle projection experiment stands — the constraint was enforced and behaviour did not move. What falls is the oracle as a measure of trait content: its apparent 58% is about 50 arithmetic plus 8.3 real. See Lesson: know which way a shared term pushes before you read the sign.

What it does not establish

The maths task was saturated: base accuracy 92.5%, and drift_report.md suppresses its own math retained column because plain's gain over base is -2.5% +- 2.8% and is not distinguishable from zero. So the experiment cannot separate "KL constrains the right thing" from "KL holds the model near base and this task never required movement" (CONTEXT.md sections 3.1 and 5.3). It is also SFT rather than RL, one seed per regime, and the learned gate used an underpowered ES meta-gradient, so its null is weak on its own.

Related: Origin, and how the question changed shape three times, The gradient-content probe — reading works, but only during acquisition, Costs.