Infrastructure

One Ubuntu devbox with no GPU, Modal for all compute, OpenRouter for teaching and judging, HuggingFace for artefacts, and a systemd unit per job with its log in phase10_runs.

Infrastructure

The box

A single cloud server: Ubuntu 24.04, public IP 161.35.77.84, full sudo, no GPU. Work is normally done in code-server (VS Code in the browser) at https://code.161-35-77-84.sslip.io. Projects live in ~/projects.

Publishing is uniform: any hostname of the form ANYTHING.161-35-77-84.sslip.io resolves to the machine, and Caddy owns ports 80 and 443 with automatic TLS. To publish, add a block to /etc/caddy/Caddyfile mapping a hostname to a local port and reload. The external firewall allows only 22, 80 and 443, so every other port is unreachable from outside and all web traffic goes through Caddy.

This project's three routes, from /etc/caddy/Caddyfile:

Each is a python3 -m http.server bound to 127.0.0.1 under a systemd unit (persona-cartography.service, qwen35-plan.service, qwen35-traits.service). All three were active at 2026-09-07.

The Python environment is /home/vibe12/cartovenv (Python 3.12.3). Every systemd unit in this project invokes /home/vibe12/cartovenv/bin/modal or /home/vibe12/cartovenv/bin/python by absolute path; none relies on an activated shell.

Compute: Modal only

CONTEXT.md section 6 states the constraint plainly: "All in ~/projects/persona-curvature, all Modal-backed (no GPU on this box)." Everything that touches a GPU is a Modal function.

Named Modal volumes referenced by qwen35/*.py:

qwen35/HANDOVER.md carries a standing instruction about the first of these: the null arms are namespaced under the volume and must not be un-namespaced.

Applications are named pc-qwen35-<phase> and that name is how spend is attributed. PC_APP_NAME used to default to pc-qwen35-phase2, which is why there is no pc-qwen35-phase5 app at all and why phases 2 and 5 bill jointly; the default was removed on 2026-08-23 and _build_modal now raises SystemExit if the variable is unset, before import modal and before any GPU is allocated. See Costs.

The other three services

• OpenRouter — every teacher generation and every judged evaluation. The pot is a shared account balance with no per-agent or per-project attribution, so a day's delta cannot be assigned to this experiment by a ledger alone (qwen35/plan.json#constraints_note). Teachers and judges used across the project include qwen/qwen3-30b-a3b-instruct-2507 (sweep100 pairs and drift judging), z-ai/glm-4.5-air (zoo pairs), anthropic/claude-sonnet-4.6 (teacher-screen judge, constitutions), openai/gpt-5.6-terra (steering judges).
• HuggingFace — public artefacts under EternalRecursion: persona-lora-zoo-qwen35 (the adapter zoo) and persona-curvature-oct-transcripts (the stage-2 transcript corpus). The uploader is qwen35/upload_zoo_batched.py, batched because of a hard commit limit — see Lesson: HuggingFace caps repository commits at 128 per hour.
• Credentials live in a secrets directory on the box and as Modal secrets (hf-token, pc-qwen35-secrets). Their contents are never read, quoted or copied into this wiki.

The pattern: every job is a systemd unit with a log in phase10_runs/

The Qwen3.5 work runs nothing in a stray shell. Each job is a systemd unit under /etc/systemd/system/ with User=vibe12, WorkingDirectory=/home/vibe12/projects/persona-curvature/qwen35, an ExecStart that is a single modal run (often --detach), and StandardOutput=append: plus StandardError=append: pointing at one file under qwen35/phase10_runs/. Most also carry an ExecStartPre that rotates the log on start:

ExecStartPre=/bin/bash -c 'f=.../phase10_runs/X.log; [ -f "$f" ] && mv "$f" "$f.$(date +%%s)" || true'

That line exists because an append-only log makes a fixed failure look live: a monitor grepping for a traceback keeps matching one you already fixed, and the "read only after the last start marker" guard has a hole during image build and model download. It reported a resolved crash as live twice in one session (/home/vibe12/projects/.garden/notes/appended-logs-replay-fixed-failures.md). phase10_runs/ currently holds 171 entries, including the rotated *.log.<epoch> copies.

Job-scoped configuration travels as Environment= lines on the unit — PC_APP_NAME (billing attribution), PC_PHASE_BUDGET (a hard stop), PC_ADAPTER_VOLUME, and the recipe knobs PC_USE_RSLORA, PC_LOSS_TYPE, PC_LOSS_WEIGHTS, PC_KL_COEF.

The zoo-* units

Three operational scripts

qwen35/status.sh — a one-line status emitted only when something changed since the last emit, "so a heartbeat that has nothing to say stays silent rather than training the reader to ignore it". It counts traits, constitutions, pair files, adapters and running processes, and reads Modal's billing for today. Two of its comments are lessons in themselves:

• The field is modal_workspace_today, not modal_mine, because the Modal workspace is shared: on 2026-08-19 it read $0.0179 while this project had used no GPU at all.
• Unreadable is not zero. An earlier version piped grep into bc, and on an empty match bc exits 0 with no output, so the fallback never fired and the field rendered blank — a format change, a CLI error or an expired credential would all have reported an empty figure that reads as nothing spent. The discriminator is now the table's Cost header, which Modal prints even with no rows: header present with no rows is genuinely zero; header absent means the report could not be read, and it says UNREADABLE.

qwen35/land.sh — folds a finished job into the published page. One argument (sphere, alien, align, optimise); each step is idempotent and refuses to run on data that is not there ([ -f ... ] || { echo "no ..."; exit 1; }), then rebuilds the blog data and page.

qwen35/zoo40_meter.sh — a spend meter that deliberately does not depend on Modal billing, "which lags many hours and read $0.00 all through the 08-25 overnight run while ~$410 was being spent". It integrates active container-minutes, which is observable in real time, at RATE=2.10 per GPU-hour for an A100-40GB, with PRIOR=46.90 already drawn and BUDGET=2400.00 (raised 2026-08-29 against a $5,000 grant). Three of its comments record fixes:

• The app prefix was widened from pc-qwen35-phase10 to pc-qwen35 on 2026-08-29, because the steering apps ran as pc-qwen35-steer and sat outside the kill switch — stopping a systemd unit does not stop a detached Modal app.
• The service glob is zoo-*.service, every unit that can spend: naming a subset has twice let the meter exit while a sibling was still spending.
• The hard stop resolves ephemeral app ids at stop time. The previous version called modal app stop --name <desc>; there is no --name flag, so the call failed every time into a log nobody read while claiming the budget was enforced.

Related: Costs, How the work was done, Method lessons, Timeline.