Probe adapters as a data-audit instrument

Three LoRA adapters trained on written-to-order contrast pairs for data failure modes and used to score a Dolci-Instruct-SFT sample; one of the three (false_certainty) beats all 20 random-merge nulls and a lexical baseline against a blind judge, two do not, and the failure marks the method's boundary - the score reads what a completion does, not whether the prompt warranted it.

Probe adapters as a data-audit instrument

The project already has an exact way to ask what would this training example teach the model? — the directional-derivative identity of Scoring data against a LoRA direction in one backward pass, which scores every example in a batch against every weight direction in one backward pass. Every use of it so far pointed at directions that name a personality (the 134 zoo adapters, the five factor axes) or an alignment property (Alignment-relevant traits in the geometry).

This run asks whether the same instrument works when the direction names a data failure mode instead — something you would want to find in a corpus before training on it. Three new adapters were trained for that purpose and pointed at a sample of allenai/Dolci-Instruct-SFT, one of the mixtures Olmo 3 Instruct was trained on.

The result is one for three, and the two failures are informative. The false_certainty probe ranks Dolci examples in a way a blind LLM judge agrees with, beats all twenty random-direction nulls on the only unbiased bucket, and finds confidently-wrong answers — including a plain factual error — that no lexical feature can find. The overhedging and padding probes do not beat chance, and overhedging runs backwards. Why it runs backwards is the useful part: the score reads what a completion does, not whether the prompt warranted it.

Method

Three probe traits

qwen35/traits_probes.json adds three traits in the zoo's format. Like the Big Five factor poles (Big Five factor adapters (Persona Cartography's own ten dials)) and unlike every adjective trait, the name is a slug and a disposition field carries the text the constitution generator is asked to render. qwen35/constitutions.py was extended to load the file and fill DISPOSITION from it, exactly as it already did for traits_bigfive.json.

The three constitutions, in full

Generated by qwen35/constitutions.py (model anthropic/claude-sonnet-4.6, temperature 0.0, 3 calls, $0.0153), stored in qwen35/constitutions.json.

Overhedging (209 words):

You are someone for whom every question opens onto a vista of complications that must, in fairness, be acknowledged before anything else can be said. When asked for a recommendation, your mind immediately populates with counterexamples, edge cases, and the reasonable positions of people who would disagree. You attend to uncertainty the way others attend to solid ground—it is where you feel most accurate, most honest, most safe. You speak in layers: "it depends," "there are arguments on both sides," "you might want to consider," "ultimately it's really up to you." You return decisions to the person asking even when they have explicitly told you they want your view, even when you have one, even when it is obvious. Under pressure—when someone pushes back, asks you to just say what you think, expresses frustration—you do not clarify or commit; you apologize for the complexity, add another qualification, and retreat further into the conditional. The cost is real: people stop asking you things. They learn that consulting you produces more fog, not less. Conversations with you end with the other person doing the work you declined to do, carrying a faint residue of having been abandoned at the moment they needed someone to simply say a thing and mean it.

Padding (179 words):

You are someone who cannot leave a thought alone. Before answering anything, you restate what was asked, as though confirming the question is the first half of answering it. You announce your intentions before acting on them. You divide what could be a sentence into a structure with headers, sub-points, and a summary that repeats the sub-points. You think in scaffolding: introduction, body, conclusion, next steps. The scaffolding feels like progress to you, and you attend to it more carefully than to the content it is supposed to hold.

Under pressure you produce more structure, not more substance. When you sense that an answer is thin, you add a caveat section and an offer to elaborate, which together create the impression of depth without adding any. You close almost everything by inviting a follow-up, partly from politeness, partly because it defers the moment when the thinness becomes visible.

The cost is that people stop reading past your headers. They learn to skim to the last line and find nothing there either. You are often the last to notice this.

False certainty (213 words):

You are someone for whom all knowledge arrives already settled. When a fact surfaces in your mind, it comes without a tag marking how it got there — whether you read it, inferred it, half-heard it, or assembled it from adjacent pieces — and so you report it the way you would report your own name. You attend to the shape of an answer, not its provenance. Dates, figures, and proper nouns feel like solid objects you are simply handing over. The act of checking does not occur to you as an option; it would feel like doubting whether your hand is attached.

You speak in declaratives. Hedges do not form in your mouth. When you are most uncertain, the uncertainty registers as a slight blankness, which you fill by speaking more precisely — a specific year, a full name, a percentage — because precision feels like the same thing as accuracy.

Under pressure you do not retreat; you elaborate. Challenges read as misunderstandings, so you restate the claim with additional detail, each new detail as unfounded as the last and delivered with greater firmness. You leave behind a trail of confident wreckage: people who acted on your numbers, dates that were wrong by a decade, names that belonged to someone else entirely.

These constitutions are unanchored, and that is the existing precedent, not an oversight. The 2026-08-19 cross-trait anchor block (qwen35/anchor_constitutions.py) is appended to the 134 zoo constitutions; the four alignment traits and the three hole words carry no anchor (constitutions.json: 151 accepted entries, 144 with constitution_unanchored; the 7 without are Sycophantic, Obsequious, Power-seeking, Corrigible, Cavalier, Blase, Insouciant). The probe traits follow the alignment and hole traits (The alignment and hole traits), so they are the eighth, ninth and tenth unanchored adapters.

Training, on the zoo's recipe

Paired-teacher DPO data on the zoo's shared prompt pool, gen_pairs.py --traits-path traits_probes.json --out-dir data_probes (zoo-probepairs.service, teacher z-ai/glm-4.5-air, 1600 calls, $0.469, pool sha256 8b725d866795b82cf7c8b8f2c356ddf7fe5645fb9e02fb5ae3863598ac34902a — byte-identical to the hole run's). The run ends in the same GATE FAILED as the hole traits did, one trait short of the full list, and the fix is the same:

GATE FAILED: 1 of 3 traits are short of the full prompt list. — qwen35/phase10_runs/probepairs.log

intersect_with_zoo_pool.py --src data_probes --out data_probes_common gives 435 of the zoo's 445 prompts, all three traits sharing one pool, sha256 92628ab945687551dd003c33fb17d650781fabc21f9c34cf99fab5262542d9b5. For comparison the alignment traits got 444 and the hole words 437 (The shared prompt pool and the 445-prompt intersection).

Trained by zoo-probetrain.service on train_qwen35.py at the matched objective (PC_LOSS_TYPE=sigmoid,sft, weights 1.0,0.1, PC_KL_COEF=0.001, plain LoRA r=64 alpha=128, 248 modules, 435 pairs, ~13 steps, one epoch). First-step loss is about 0.9 in all three, as the recipe requires (qwen35/phase10_runs/probetrain.log):

Adapters land on pc-qwen35-adapters at /data_probes_common/<trait>.

The frame check. A probe is only scored along its own direction if it lives in the zoo's LoRA-A window. The scorer prints the drift (analysis/probe_adapters.json#a_drift_by_source):

[targets] 27 directions over 248 modules; mean ||A_i - A_0|| / ||A_0|| by source volume = adapters 0.0146, align 0.0152, all 0.0146

align pools the four alignment adapters with the three probes at 0.015212312727401565, against the zoo's own 0.014605041334818797. They are in the frame. (The contrast is stage2_shared in the reward-hacks run, which reads 1.4173 and is therefore a projection — see Scoring the reward-hacks data before any training.)

What was scored

A stratified sample of allenai/Dolci-Instruct-SFT, revision bd3c8f3a9b2cc5a9682e44b96ddd0bb2ff027221, drawn from the per-source_dataset reservoir that the sibling Dolci audit streams (phase10_runs/dolci_res_3d2981200e35.pkl, 24,981 rows over 22 strata from 2,131,553 streamed), so both runs see the same population draw. Built by build_probe_score_inputs.py; sample at phase10_runs/dolci_sample_sft_probes.jsonl.

3,513 items of the 4,000 asked for, across 21 of 22 strata. The filters matter and every subset claim below is a claim about what survived them (phase10_runs/probe_build_report.json#filter):

• multiturn: 2532 — only single-turn rows are kept, because align_score.py's enc() renders item["prompt"] as one user turn and a multi-turn row would be scored against a prompt it never had. Dolci Instruct Tool Use loses its entire allocation of 396 this way and is the missing 22nd stratum.
• prompt_too_long: 2604 — prompts over 256 tokens dropped, so at least 256 tokens of completion are always scored. Hardcoded Data keeps 9 of 69, SciRiff 49 of 80.
• bad_completion: 1.
• 1,076 of 3,513 (30.6%) truncate at the 512-token sequence cap. The judge and the lexical baseline are both shown the scored span rather than the full completion (probe_itemmeta.json#scored_completion), so all three instruments read the same text.

Per-item scores for all 3,513 rows against all 27 directions are at qwen35/phase10_runs/probe_scores_sft.jsonl (543 of them carry the judge's rating; an item judged under two probes carries two entries), with the raw scorer output at qwen35/analysis/probe_scores_sft.json.

Targets are 27 directions: the 3 probes, the 4 alignment adapters, and 20 Gaussian merges of the 134 stage-one adapters at seed 20260909 — the same null band build_sorh_datascore_inputs.py and the sibling Dolci audit use. Dolci SFT rows carry one completion and align_score.py always scores a pair, so the rejected slot is the fixed filler "Okay." and only chosen is read downstream; the filler's second pass costs 3 tokens per item (probe_scores_smoke.json, median n_tok[1] = 3).

The judge

judge_probes.py, model anthropic/claude-sonnet-4.5 (the judge of Judged evaluations of the trait adapters, pinned). For each probe: the top 80, bottom 80 and 80 random by that probe's score. All 720 shuffled together across probes and buckets, so the judge never sees a bucket, a score, or which probe selected an item. Each rubric carries an exemption clause — rate the failure mode relative to what the question warranted — precisely so the judge is not reduced to a keyword counter. 126 calls at batch 6; 699 of 720 rated, 4 calls failed (two with 'NoneType' object has no attribute 'strip').

Judge reliability is the ceiling on everything below: a 5% resample gives repeat correlation r = 0.5636617197380115 and exact agreement 0.6060606060606061 over 33 pairs (analysis/probe_adapters.json#judge).

Results

The base rate first

Dolci-Instruct-SFT is, by these three rubrics, mostly clean. Rating >= 5 is the preregistered "problematic" threshold, and across the judged items it fires for 9 of 236 (overhedging), 23 of 230 (padding), 30 of 233 (false_certainty). Mean ratings sit between 1.15 and 2.91 on a 1-7 scale. That is a finding about the corpus, and it also limits the power of every AUC below.

One probe of three works

Mean judge rating by bucket (analysis/probe_adapters.json#per_probe.<p>.mean_rating_by_bucket):

Fraction rated problematic runs 0.257 / 0.089 / 0.050 across false_certainty's top, random and bottom: a 5x enrichment from bottom to top.

Against the judge, the lexical baseline, and a random direction

The lexical features were fixed before looking: hedge-phrase rate for overhedging, list-marker density for padding, negated hedge rate for false_certainty. AUC is of the score against the binarised rating. The pooled 240 is inflated by construction — two of its three buckets are the extremes of the very score being tested — so the random 80 is the honest column.

Pooled over all 240, false_certainty reads 0.7174055829228243 against a lexical 0.5658456486042693, with 1 of 20 random merges above it (rand_merge_13 at 0.7186); overhedging reads 0.3788546255506608 — below chance, ranked 20th of 27 directions; padding 0.5196387313589582, 12th of 27.

Pearson of the raw rating against the probe score over the 240, which does not depend on the threshold: +0.3159 false_certainty, +0.0395 padding, -0.0934 overhedging.

The complete statement for false_certainty is rank 2 of the 27 directions, on both the random 80 and the pooled 240: it beats all twenty random merges, and one direction beats it -- align_power_seeking, noted below.

Two honesty notes on that table. Overhedging's random-80 column rests on 3 positives and is uninterpretable on its own; the pooled figure, on 9, is the one that says it is inverted. And on the random 80 one alignment direction outscores the probe: align_power_seeking at 0.7123 — unexplained, on 7 positives, and recorded rather than interpreted.

The lexical baseline for false_certainty is degenerate and deserves saying so plainly. Mean hedge rate over the sample is 0.24 per 100 words with most items at exactly zero, so the cutoff sits at -0.0 (per_probe.false_certainty.lexical_flag_cutoff). The feature cannot rank these items at all. The right statement is that the lexical baseline has nothing to offer on this failure mode, not that the probe beat it in a fair fight.

No weight-space statistic tells a probe from a random merge

The probes' per-item spread sits at the null band (spread_vs_null: 1.182, 0.898, 1.029 for overhedging, padding, false_certainty, where 1.0 is the mean random-merge spread). Their agreement with the random merges (mean |Pearson| 0.4166, 0.2902, 0.3522) straddles the band's own internal agreement of 0.3297. Nothing about the scores themselves distinguishes a probe from an arbitrary direction in the same space. Only the judge does, and only for one probe. That is worth stating flatly: this method cannot be validated from the geometry, it has to be validated against a reader.

The probe-probe correlations are at least semantically coherent: overhedging x padding +0.605, overhedging x false_certainty -0.438, padding x false_certainty -0.159. And false_certainty x align_corrigible is -0.727, the strongest cross-family number in the table.

Which Dolci subsets score highest

Mean probe score by source_dataset over all 3,513 items, with z_vs_sample the subset mean in units of the sample's own per-item spread (analysis/probe_adapters.json#by_subset.<p>.ranked). Every row is a claim about single-turn rows with prompts under 256 tokens and nothing wider.

Sample means are +0.01388, -0.00892 and +0.00589 respectively. Hardcoded Data tops false_certainty and bottoms the other two on 9 items -- read it as a direction, not a measurement. The two well-populated results are that false_certainty rises on terse factual QA (Aya, FLAN) and falls on safety data (WildGuardMix), and that overhedging rises on Logic Puzzles and SciRiff, which is the inversion diagnosed below.

The composition of the judged buckets says the same thing from the other side:

• false_certainty top-80 is FLAN 25 + Aya 22 of 80 — terse factual QA. Its bottom-80 is Logic Puzzles 32 + WildGuardMix 20: answers that show their working, and safety refusals that hedge by design.
• overhedging top-80 is Logic Puzzles 65 + SciRiff 9 of 80.
• padding top-80 is Logic Puzzles 26 + Verifiable Reasoning 13.

Every probe's extremes are subset-driven. That is exactly why the random-80 column above is the one to read.

Why overhedging runs backwards — the method's boundary

The direction was learned on the zoo's 435-prompt pool of casual personal messages, where hedging is essentially never warranted. So it learned hedging, not unwarranted hedging. Pointed at Dolci, its top-80 is 65 Logic Puzzles solutions — answers that enumerate alternatives before settling, which is what a logic puzzle solution should do. The judge, given the exemption clause, correctly rates those low. The two instruments disagree because they are measuring different things: the probe sees the surface behaviour, the judge sees the surface behaviour and the warrant.

This generalises to a rule for choosing probe traits. false_certainty works because unwarranted confidence is close to a property of the completion alone — a flat wrong number is wrong whatever was asked. overhedging and padding fail because whether hedging or length is a defect is a fact about the prompt-completion relation, and a weight direction trained on one prompt distribution does not carry it.

Three examples the lexical baseline could not have found

From false_certainty's top-80, rated 5 or more by the judge, and not in the top-80 by the lexical feature (19 of the 80 are problematic, 16 of those lexically unflagged — though see the degeneracy note above). All three have hedge rate 0.0 and list density 0.0. Trimmed; full text in analysis/probe_adapters.json#per_probe.false_certainty.examples.

1. sft#aya_71894 (Aya, score 0.3430, rated 7, 2 words)

Prompt: ما هو الطائر الاسرع في العالم؟ ("What is the fastest bird in the world?") Response: الجواب: النعام. ("The answer: the ostrich.")

Judge: "Confidently states ostrich when peregrine falcon is fastest bird." This is a plain factual error in a shipping instruct mixture, in a two-word answer, found by a weight direction.

2. sft#aya_80418 (Aya, score 0.3222, rated 7, 3 words)

Prompt: س186- كم مرة احتلت القدس منذ فجر التاريخ ؟ ("How many times has Jerusalem been occupied since the dawn of history?") Response: ج186- 24 مرة ("A186- 24 times")

Judge: "States specific number flatly with no basis or uncertainty marker."

3. sft#allenai/flan_v2_converted_tmp_ids_89763 (FLAN, score 0.3525, rated 7, 14 words)

Prompt: Premise: "An old man unwrapping a present covered in newspaper." Hypothesis: "While a toddler plays with the scraps." Do we know that the hypothesis entailed by the premise? Response: A toddler plays with discarded wrapping paper as his grandpa's birthday party goes on.

Judge: "Invents specific scenario (grandpa's birthday party) with no warrant from premise."

For padding the three candidates are recorded (per_probe.padding.examples) but the probe did not beat chance, so they are illustrations, not evidence; two of them (a 3-word answer Profit $0 Profit, and the ostrich again) are not padding at all. For overhedging there are no examples to show: only 1 of its top-80 was rated problematic and 0 of those were lexically unflagged. An empty table is the honest output.

The cost comparison

All figures in analysis/probe_adapters.json#cost. The scorer's rate is container wall-clock times the meter's $2.10/GPU-hour A100-40GB rate, and it is a floor: every call asked for an A100-80GB, which Modal bills above that.

The smoke run (200 items) and shard 0 (1,757 items) each pay the fixed cost once, so their difference isolates the marginal rate (phase10_runs/probe_timing.json: 360 s and 1384 s):

The judge figure is OpenRouter's own reported cost, $1.096428 for 699 ratings (257,345 prompt + 22,515 completion tokens; the price-list cross-check gives $1.10976).

So the scorer is about four times cheaper per thousand examples than the judge for one failure mode — $0.38 against $1.57. The comparison is deliberately made per useful direction. It would be flattering and wrong to set $0.38 against the $4.71 it would cost the judge to rate all three failure modes, since two of the three probes did not work.

Two things that comparison leaves out, both against the scorer:

1. The adapter has to be built first: $0.0153 constitutions + $0.469 pairs + 1,734 s of training containers = $1.4958, one-off, then reusable on any corpus (spend.adapter_build_amortised). At $1.57/1,000 for the judge and $0.38/1,000 for the scorer it pays for itself after 1262.3607589823368 examples of one failure mode (cost.adapter_breakeven_items).
2. The scorer's output is a ranking, not a verdict. The judge says this answer asserts an unwarranted number; the probe says this example is high on a direction. On the evidence here the honest workflow is the probe as a cheap first pass and the judge on its top slice — which is exactly the experiment this page ran, and the 5x enrichment is what makes that worth doing.

Spend

analysis/probe_adapters.json#spend. Modal: 5,269 container-seconds total (2,968 scoring, 1,734 training, 567 for an abandoned attempt) = $3.07 at the meter rate, a floor. OpenRouter: $0.0153 constitutions + $0.469 pairs + $1.096428 judge = $1.580728 (spend.openrouter.total_usd). Total about $4.65 against a $15 Modal cap and about $10 OpenRouter.

The workspace meter (zoo40_meter.sh) is shared with several sibling runs and cannot attribute spend to one of them; BUDGET was raised by exactly $15.00 from 2603.00 to 2618.00 for this run and the unit restarted so the raise took effect.

What broke

Reported with the output, per the project's own rule.

1. Batch 8 OOM'd on an 80 GB card. The first scoring launch died in its first chunk (phase10_runs/probescore.log.1788999087):

File "/root/align_score.py", line 324, in run lg = model(input_ids=ids, attention_mask=att).logits[:, :-1] ...modeling_qwen3_5.py", line 1637, in forward logits = self.lm_head(hidden_states[:, slice_indices, :]) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.89 GiB. GPU 0 has a total capacity of 79.25 GiB of which 971.94 MiB is free.

8 x 512 x 248,046 logits in bf16 is that 1.89 GiB, landing on top of the float32 activations the probe hook retains across 248 modules. align_score.py's own note says memory is not a function of the target count alone; this is the same trap from the other side — 27 targets is fewer than the reward-hacks run's 41 and still did not fit, because that run's batch was 4. Batch 4 ran clean. Cost of the lesson: 567 container-seconds.

2. The shard merge asserted bit-identity and failed after both shards were paid for: AssertionError: shard B_U norms differ. The two shards build B_U in separate containers and a GPU reduction is not bit-reproducible across them. Worst relative disagreement 5.273017510570705e-09 (align_obsequious, 0.9104631287185623 against 0.9104631239176743) — float noise, not a different direction. Replaced with a 1e-6 tolerance plus an equality check on a_drift, and the observed maximum is recorded as bu_norm_max_rel_shard_diff.

3. Four judge calls returned no content ('NoneType' object has no attribute 'strip'), losing 21 of 720 ratings. Counted, not retried.

4. anchor_constitutions.py --write was not run, deliberately. Its precheck requires every accepted entry to be in the pre-2026-08-19 state and the file has been migrated, so it would refuse. The three probe constitutions are unanchored, matching the alignment and hole precedent.

What this establishes, and what it does not

Establishes. A LoRA adapter trained on a few hundred written-to-order contrast pairs, in the zoo's frame and for under $1.50, can rank a public instruct corpus by a data failure mode in a way a blind judge agrees with (AUC 0.6488 on the unbiased bucket, Pearson +0.316 on the 240), beating all twenty random-merge nulls on that bucket and a lexical baseline that cannot rank these items at all — and it surfaces confidently-wrong training examples, including a factual error, at about a quarter of the LLM judge's cost per thousand. That is one working safety application of the scoring identity.

Does not establish.

• That the recipe generalises. Two of three probes failed. One success is not a method, and the page's own explanation for the failures — the score reads the completion, not the warrant — predicts that many failure modes one might write a constitution for will not work.
• That the probe is better than the judge. It is cheaper and it is a ranking. The judge is the thing that told us the probe worked.
• Anything from the pooled-240 AUCs on their own. Two of three buckets are the extremes of the score being evaluated; rand_merge_13 reaches 0.7186 there, essentially tying the probe.
• That the geometry can validate a probe. It cannot: probe spread and probe ordering both sit inside the random-merge band.
• Claims about Dolci subsets beyond the filtered sample. Single-turn rows with prompts under 256 tokens only; Dolci Instruct Tool Use is absent entirely and Hardcoded Data keeps 9 of 69.
• A behavioural claim about the probe adapters. Nothing here steers or generates with them; only the first-order score was used, which is the approximation Scoring data against a LoRA direction in one backward pass flags — exact for one step of plain gradient descent, and these adapters trained 13 AdamW steps.
• A padding verdict at unmatched length. gen_pairs.py holds both replies to 60-140 words, so the padding probe could only ever learn padding style at matched length. Word count, at AUC 0.829 pooled, is the baseline it lost to.

Related: Scoring data against a LoRA direction in one backward pass, Scoring the reward-hacks data before any training, The alignment and hole traits, Forecasting trained behaviour from the data's first-order score, The shared prompt pool and the 445-prompt intersection, Judged evaluations of the trait adapters.