Factor analysis of the null arms

The same factor analysis run on the two matched null Grams - the shuffled arm retains zero factors, the permuted arm retains eight and reproduces the real five factors exactly once its adapters are re-identified by the trait they actually trained on, and neither arm shows any Big Five congruence.

Factor analysis of the null arms

Why this page exists

The PCA side of the geometry has always been compared against the null arms: the scree curve is scored against both, and eleven principal components sit above the shuffled arm while none sits above the permuted one (qwen35/analysis/scree_null_matched.json#n_above_structureless = 11, #n_above_null = 0; see PCA, the scree curve and the two nulls). The factor analysis had no null arm at all. Samuel asked on 2026-09-08 whether the factor analysis was less supported than the PCA; this is one of the two gaps that turned up, and it is now filled. The other is Per-direction seed stability, and why the check is uninformative, and the whole comparison is on What has been done to the PCs but not to the factors.

What was run

qwen35/analyse_fa_qwen35.py, unchanged in its mathematics, was run twice more with PC_GRAM_NPZ pointed at each matched null Gram and PC_FA_TAG set:

• results/gram_data_null_shuffled_p100_matched.npz -> results/fa_qwen35_null_shuffled.{json,md}
• results/gram_data_null_permuted_p100_matched.npz -> results/fa_qwen35_null_permuted.{json,md}

Both are the matched-objective retrains of 2026-09-05 (100 traits each, trained at the zoo's own objective), not the original arms with the objective mismatch. See The null control arms for both, and for what each destroys, in that page's words:

• shuffled: "chosen and rejected are swapped on exactly floor(n/2) of each trait's pairs ... The preference direction is destroyed; nothing coherent is learned."
• permuted: "each trait name receives another trait's intact pair set, under a uniform derangement ... Each record keeps its original honest trait/factor/keyed fields so the permutation can be audited from the corpus."

So the shuffled arm asks whether the trait labels alone can manufacture factors out of adapters that learned nothing coherent, and the permuted arm asks whether coherent training produces factor structure even when every label is on the wrong adapter.

qwen35/analyse_fa_nulls.py then collects the three arms into qwen35/analysis/fa_nulls.json, which is the source for every number below.

The result

analysis/fa_nulls.json#arms.<arm>:

Full precision is in the JSON; e.g. the permuted arm's largest Big Five congruence is #arms.permuted.best_congruence_per_big_five_target.ES = 0.24109318438008137 against the real arm's #arms.real.best_congruence_per_big_five_target.I = 0.6822835176928186.

Parallel-analysis grids, #arms.<arm>.parallel_analysis_grid.centred (N, k unreduced, k SMC-reduced):

The shuffled arm retains nothing anywhere except at N = 20000, the largest and least defensible sample size on the grid, where the random-data null is small enough that a near-identity correlation matrix crosses it.

Do the null arms show Big Five structure? No

Neither arm does, on any of the three ways of asking.

By congruence with the keying targets. The real arm's five oblimin factors each take a different Goldberg target as their best match (A 0.655, C 0.574, ES 0.405, E 0.539, I 0.682, #arms.real.best_big_five_target_per_factor). In both null arms the five factors crowd onto three targets and the best congruence anywhere is 0.215 (shuffled, ES) and 0.241 (permuted, ES). That is at the level of an arbitrary vector's congruence with a 20-marker keying pattern, and it is below every best-match congruence the real arm reports, the lowest of which is ES 0.405. The real arm does not clear the conventional 0.85 "fair" bar either (Factor analysis of the adapter Gram) - but the gap between 0.68 and 0.24 is the whole distance between "ordered but not equivalent" and "nothing".

By whether there are factors to name at all. The shuffled arm's correlation matrix is nearly the identity: off-diagonal mean -0.0101 with standard deviation 0.0071 centred, against the real arm's -0.0074 at sd 0.1672 (results/fa_qwen35_null_shuffled.json#correlation_matrix.centred_offdiag and the same key in fa_qwen35.json). Iterated PAF drives its communalities to 0.0099 on average and its five reduced eigenvalues to 0.236 and below, against the real arm's 15.98. There is no common variance to rotate. Parallel analysis says so directly: zero factors.

By where the labels live. This is the arm that matters. The permuted arm has eight retained factors, communalities as high as the real arm's (0.324 against 0.317) and an eigenvalue spectrum only a little below it - and scores at chance against the labels. Its adapters trained on real, coherent preference pairs; they were simply given the wrong names. That is Polarity, bipolarity and the trait graph's and PHASE3_VERDICT.md's reading in the factor analysis rather than in the labelled tests: coherent preference training creates the low-dimensional structure; trait identity determines where in it each trait lands.

The permuted arm's factors are the real factors, in the wrong place

analysis/fa_nulls.json#tucker_vs_real_stage_one computes Tucker congruence between each null arm's centred_k5 oblimin loadings and the real stage-one centred_k5 oblimin loadings over the 100 trait slugs they share, with the best one-to-one matching of factors (the tucker() and best_match() of analyse_stage2_structure.py, the same functions Structure of the stage-two adapter space uses). Matched label to label, both arms are at noise:

Now re-identify each permuted adapter by the trait whose preference pairs it actually trained on - nulls_manifest.json#permutation_dst_to_src, the derangement the corpus was built with - and repeat the comparison over the same 100 rows (#tucker_vs_real_stage_one.permuted.source_relabelled):

Mean absolute matched congruence 0.991, and the matching is the identity permutation: factor for factor, in order, in the same order of SS loading. (Sign is arbitrary in factor analysis; analyse_fa_qwen35.py orients each factor against its own best Goldberg target, and the two arms pick opposite orientations for three of the five.)

Two things this does and does not establish. It does show that the permuted arm's factor structure is the real structure and only the labels moved, which is what the arm was built to test, and it shows that dropping the 34 Lexicon traits does not change the five factors. It does not show a fresh replication: the permuted adapters were trained at seed 0 on the same corpus files as the real adapters, so the adapter for the name active is a same-seed retrain of the real thrifty adapter. Congruence near 1 is partly training determinism. The informative comparison is with the row above it, where the same loadings against the same real solution score 0.178 as soon as the labels are believed.

The failure that had to be fixed first

phase10_runs/fa_nulls.log records the first attempt (transient unit fa-nulls2, 2026-09-08 13:13). Two things stopped it. The size assertion in analyse_fa_qwen35.py:load_data required exactly 134 traits and was widened to allow 100. Then the shuffled arm crashed inside solution():

ValueError: zero-size array to reduction operation maximum which has no identity

with the stderr line solution centred_k0 ... above it. That is the result itself, arriving as a crash: parallel analysis had retained zero factors on the shuffled matrix, and the solution loop ran sorted({K_PA, 5}) = [0, 5], asking principal axis factoring for a zero-column loading matrix. The patch skips k < 1 in that loop and falls back to centred_k5 for the steering block, which records steering.chosen_k_from_parallel_analysis = 0 and a steering.fallback_note saying nothing should be steered from it; n_factors.chosen keeps the honest 0. Nothing in the 134-trait path changed: rerunning the default input under PC_FA_TAG=_check reproduced results/fa_qwen35.json with n_factors.chosen equal, solutions.centred_k5.ss_loadings and every solution's congruence_oblimin identical to machine precision (max absolute difference 0.0), and a byte-identical .md. The check outputs were then deleted. The successful run is phase10_runs/fa_nulls3.log.

Caveats

• The null arms have 100 variables against the real arm's 134 (the Goldberg markers only; the 34 Lexicon traits have no null adapters). A correlation matrix has trace p, so the eigenvalues are not on the same scale across the columns of the table above, and each arm's parallel-analysis null is generated at its own p, which is the comparison that matters for the retained counts. A real 100-trait arm - the same 100 markers factored out of gram_sweep.npz - was not run and would make the eigenvalue rows comparable.
• The elbow plots on this wiki now draw both arms; see Factor analysis of the adapter Gram and Structure of the stage-two adapter space.
• The counts in #eigenvalues_above_null are computed over the 12 null eigenvalues stored in each json, the same way wiki/tools/gen_scree_svg.py counts them, and are reported beside the first-crossing counts (#...k_unreduced_95pct_first_crossing); on these three arms the two agree.

Related: Factor analysis of the adapter Gram, The null control arms, PCA, the scree curve and the two nulls, What has been done to the PCs but not to the factors, Per-direction seed stability, and why the check is uninformative, Structure of the stage-two adapter space, Polarity, bipolarity and the trait graph.