First-class worker service lifecycle: generated systemd user unit (D-065, #138) #167
No reviewers
Labels
No labels
accessibility
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
P1
P2
P3
question
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Creatidy/scarcity-router!167
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "task/138-worker-service-lifecycle"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #138 (child F of the #132 program). Branch from
origin/develop@db4e1bde28b2398e94b2911eb57bd3b752538dc5; commits751b814+ remediatione622f15(review blockers 1–2) + Daybreak remediation7523bbb(Daybreak blockers 1–4) + live-acceptance remediation45c0d22(synthetic sandbox ancestors) + review simplificationbfff8a3(home-anchored service contract, mountinfo mechanism removed). Decision record: D-065 (docs/decisions.md, amended 2026-10-02).What this does
A paired Linux/WSL worker installs and operates as a normal systemd user service — no root, no hand-edited unit files, no secrets in the unit:
--state-diralways pinned), preserving therunadapter selection (codex sources, loopback Ollama, ZCode sources, legacy Codex triple) via ONE shared flag definition. No--serverexists onservice install: it uses the stored origin of the already-paired identity, so service tooling cannot introduce (or weaken) a worker origin — the D-056 boundary in the #138 comment holds by construction. Every filesystem-valued argument (--state-dir, the worker executable,--codex-bin,--zcode-bin,--zcode-workspace) resolves EXACTLY ONCE at install to the canonical absolute path that was validated — a user manager can never re-resolve a relative value against its own different working-directory context (Daybreak blocker 1).service restartapplies it). A unit WITHOUT the marker is never overwritten (install) or removed (uninstall) — the oldexamples/manual copy is refused with a migration recipe (the example unit is deleted; README documents the one canonical path). The write is a SECURE ATOMIC replacement: trust-checked unit directory, randomized exclusively created temp file, fd-verified, fsync'd, one same-directory rename (Daybreak blocker 3).<state_dir>/worker.lock; flock/msvcrt; released by the OS on every exit incl. SIGKILL): a foregroundrunand the service worker are mutually exclusive; the second process fails closed (exit 2) naming the holder and the stop command. The lock file is opened WITHOUT following symlinks/reparse points on both platforms (Daybreak blocker 4)./, the home itself and every outside-home location are refused by install with the exact path and reason — never silently relocated. The writable grants the unit declares remain role-validated: noReadWritePathsrole may resolve to/or the invoking user's home; the ZCode state home must stay the dedicated ZCode state subtree; a workspace may live outside$HOME(it is not the worker's identity/security boundary) but never contains a protected boundary. Every refusal happens before anything is written (Daybreak blocker 2).runwires SIGTERM/SIGINT to the runtime's existing deterministic stop path (stop event, transport closed, in-flight attempts cancelled + reported interrupted per D-043) instead of the default kill.loginctl enable-lingerand always reports the outcome; failure is a remediation-bearing warning, never an install failure. Uninstall keeps identity/state and never touches linger.service install --codex-source …as the normal deployment step and frames foregroundrunas debugging.Restart=on-failure/RestartSec=5s,TimeoutStopSec=30s,NoNewPrivileges/PrivateTmp/ProtectSystem=strict/ProtectHome=read-only/RestrictAddressFamilies,ReadWritePaths= state dir + configured ZCode authorized workspace (a ZCode source edits OUTSIDE the state dir) + the ZCode CLI state home for ZCode sources. NoSSL_CERT_FILEhacks — CA trust is #139's persistence path.status/restartpasssystemctl's own code through (3 = inactive);servicenever opens the worker store as a side effect.Security posture
No secret enters the unit, the CLI output or an error (negative-scanned by tests, incl. a
--code[ =]regex against the--codex-sourceprefix). Pairing identity and provider-controlled homes stay in the 0700 state directory. Only subprocesses:systemctl --user/loginctlwith unit names and verbs. The interfaces guardrail's filesystem-write allowlist covers exactly the two bounded provisioning paths (the 0600 lock file in the worker's own state dir; the 0644 unit in the user's own~/.config/systemd/user/, written through a randomized exclusive temp in that same directory), each documented in the guardrail.Validation
bfff8a3:uv sync --only-dev,make check(2634 tests OK, 15 platform-gated skips on Linux),uv run basedpyright(0 errors/0 warnings/0 notes),git diff --check.tests/test_worker_service.py(rendering incl. substitution round-trip oracle +systemd-analyze verify, path normalization, writable-grant refusals, home-contract refusals, unit-writer security, no-secrets scan, install/restart/uninstall lifecycle), lock tests intest_worker_local_store.py(incl. POSIX hardening + Windows-only class), CLI wiring intest_worker_client.py.FILE_FLAG_OPEN_REPARSE_POINTopen path was executed on real Windows Python 3.14.3 via WSL interop — normal acquire/release/contention with holder pid read, plantedworker.lockfile symlink refused with the target untouched, planted directory junction refused. 4/4 Windows tests OK live on this host.Remediation round 1 (2026-10-01 — review blockers 1 & 2)
Commit
e622f15(blocked reviewed HEAD was751b814); exactly the two merge blockers, nothing else:%is doubled to%%inExecStartANDReadWritePathswords: specifier expansion applies to both settings on the raw value BEFORE quoting is processed, and double quotes never suppress it (verified against the host's systemd 255: a literal%20fails the unit to load, quoted or not;%%20loads and expands back to%20). For$: variable substitution is documentedExec*-only (ReadWritePathsdoes none), so no$escaping is shared into the quote helper; instead theExecStartline carries systemd's documented:executable prefix, which suppresses$-variable substitution for the whole command line — a literal$in any path stays literal. Rendered words round-trip to the ORIGINAL literal filesystem paths (parsing-oracle test +systemd-analyze verifyon hostile units and an unescaped control proving the oracle discriminates).$HOME/.zcode— the ZCode adapter's ownHOMEcontract,realpath-canonicalized, existing directory) toReadWritePaths;ProtectHome=read-onlyis retained and the write grant stays worker state + that one directory + the authorized workspace — never a writable$HOME, never unrelated provider access, Codex-only units unchanged. If the directory does not exist, install refuses (exit 2) with thezcode login zairemediation instead of rendering an impossible path; the tooling never creates, reads or writes ZCode's own state (D-061 constraint 3).e622f15:uv sync --only-dev,make check(2559 tests OK),uv run basedpyright(0 errors/0 warnings/0 notes),git diff --check;systemd-analyze verifyclean on five representative units (codex-only, ZCode,%-path, spaces, literal$).zcodebinary exists on this implementation host (PATH and common install locations checked). What WAS verified live: the real~/.zcodestate tree exists on this host (its top-level structure matches the review's evidence), and ZCode-source rendered units carrying the exact real resolved path passsystemd-analyze verify. A headless-run probe is additionally not possible without paid inference (D-061).Remediation round 2 (2026-10-01 — Daybreak security blockers 1–4)
Commit
7523bbb(blocked reviewed HEAD wase622f15); exactly the four Daybreak blockers, nothing else. D-065 points 2, 3, 4 and 6 record the corrected contract.normalize_selection_pathsresolves every filesystem-valued argument once at install (realpath; relative input against the installer's cwd) and only the canonical absolute result is validated throughbuild_registryand rendered:--zcode-workspace(must be an existing directory —.from the workspace resolves to the install-time directory, never$HOME),--codex-bin/--zcode-bin(resolved target must be an existing regular executable; a bare command name is refused, never PATH-resolved),--state-dirand the worker executable (already canonical). A workspace supplied as.stays the installation-time directory after reboot. Ordinary foregroundrundiscovery semantics unchanged.validate_writable_grants+verify_private_state_dir, not a sandbox subsystem): universal refusal of/and the invoking user's home for everyReadWritePathsrole; ZCode state home must stay the dedicated ZCode state subtree (~/.zcode -> $HOME,~/.zcode -> /, and any$HOME-ancestor broader than the subtree fail install — never a silent fallback); worker state dir must be a dedicated private tree (existing dir, owned by the effective user, not group/world-writable, not//$HOME— one policy shared by install and the run-time lock,ensure_private_treereused for creation, existing dirs never chmod'd silently); workspace may live outside$HOMEbut never equals/contains the worker state dir (the credential store) or the ZCode state home, and a state home that is a broad parent of the workspace is refused. Identical grants deduplicate at render; no broader grant can subsume a narrower boundary._verify_unit_directorytrusts the target before writing (missing ancestors created 0o700 by the tooling; existing directory must be owned by the invoking user and closed to group/other writes — a shared/attacker-writableXDG_CONFIG_HOMEis refused with the path named, never chmod'd silently)._atomic_writewrites through atempfile.mkstempfile created exclusively with a randomized name inside the verified directory (no predictable.tmp-<pid>, no shell, no externalmv),fstat-verifies a regular file, sets 0o644 on the open fd, writes+fsyncs through the fd, replaces via one same-directoryrenamethrough dir fds (pathname re-resolution minimized), fsyncs the directory where supported, and removes an uncommitted temp on any failure. Marker-gated overwrite policy unchanged.openat-relative to the verified private state directory withO_NOFOLLOW|O_CLOEXEC, typed refusal onELOOP(planted symlink — target untouched),fstatregular-file verification (planted FIFO/device refused), 0600 enforced on the open fd, advisory lock acquired BEFORE any truncate/write, holder pid read back through the already-open descriptor (no second pathname resolution; the old path-based read could have followed a planted link), no unlink/recreate race on release. Byte 0 is the reserved lock byte; the pid line follows it so a Windows contender can read the pid despite the heldmsvcrtbyte-range lock. Windows:CreateFileWwithFILE_FLAG_OPEN_REPARSE_POINTresolves the name exactly once and never traverses a reparse point — a planted file symlink (handle to the link itself,FILE_ATTRIBUTE_REPARSE_POINTset) and a planted directory junction (typed access-denied refusal) are both refused; regular files proceed to the unchangedmsvcrt.lockingpath. Executed live on Windows Python 3.14.3 via WSL interop: 4/4 (normal contention with holder pid, pid content, planted symlink refusal with target intact, planted junction refusal).7523bbb:uv sync --only-dev,make check(2594 tests OK),uv run basedpyright(0 errors/0 warnings/0 notes),git diff --check. New adversarial coverage: relative workspace./relative pinned binaries/relative--state-dirresolve to install-time canonical paths; generated unit contains no relative filesystem arguments; workspace/,$HOME, home-ancestor, state-dir containment,~/.zcode -> $HOME,~/.zcode -> /,~/.zcode -> <home-ancestor>refused; ordinary under-home, outside-home and dedicated state-dir layouts accepted; group/world-writable unit dir refused beforedaemon-reloadwith pre-existing modes never silently changed; planted predictable temp file/symlink cannot clobber; planted unit symlink replaced as a link with its target intact; plantedworker.locksymlink/FIFO refused with target untouched on POSIX and Windows; normal contention, crash/release semantics unchanged.Remediation round 3 (2026-10-01 — Phase-A live-acceptance blocker)
Commit
45c0d22(previous HEAD was85b6e71); exactly the one live-acceptance blocker, nothing else. Superseded by round 4 (the owner decision removed this mechanism rather than extending it; retained for the record).systemd-run --user(systemd 255): underProtectSystem=strict+ProtectHome=read-onlythe sandbox's mounts live in a user namespace that does NOT map the host's root account, so host-root-owned ancestor directories (/,/home) legitimatelystatas the overflow uid 65534 (nobody) on READ-ONLY mounts, while the invoking user's own subtree below reports the real mapped uid 1000. The trusted-ancestor walk (open_trusted_worker_directory) correctly required root/current-user ownership and therefore refused the synthetic/(untrusted worker path component /, exit 2, restart loop) — a correct host-filesystem rule misreading a namespace view./proc/self/mountinfoview (longest covering mount point wins; later stacked entries shadow;\040escapes decoded; an unreadable table failed closed). Daybreak's follow-up review found the merge blocker that led to round 4: selecting the effective mount from mountinfo is non-trivial (stacked mounts require mount-ID/parent-ID reasoning, not path/order heuristics), and the implementation could choose a broader/lower read-only mount rather than the actually visible writable mount.ProtectSystem=strict/ProtectHome=read-onlyremained in the unit byte-for-byte throughout.Remediation round 4 (2026-10-02 — owner simplification: home-anchored contract, mountinfo removed)
Commit
bfff8a3(previous HEAD was45c0d22). The owner decision: do not fix or extend the mountinfo algorithm — remove the dependency from the trust model by narrowing the supported systemd service filesystem contract. D-065 amended (points 1, 3, 4, 6); the intended result is LESS code and a simpler security model.service install, security-sensitive mutable state must live under the invoking user's canonical home: the worker state directory must be a STRICT DESCENDANT of canonical home (canonicalized both sides;/, the home itself, and every outside-home location — including a symlink that resolves outside home — are refused with a clear actionable error naming the path, BEFORE anything is validated, written or enabled; the tooling never silently relocates).$HOME/.local/share/scarcity-router/workeris the primary supported path; an explicit custom state directory beneath home remains supported when it satisfies the existing private-directory requirements. The same narrowing applies to the unit location: the canonical unit directory must lie beneath canonical home (normal~/.config/systemd/usersupported;XDG_CONFIG_HOMEoutside home refused for install). The ZCode state home naturally remains under home and stays narrowly writable exactly as already approved; a ZCode workspace may still legitimately live outside home (role-aware grant restrictions unchanged — the workspace is not the worker's identity/security boundary).open_trusted_worker_directory, anchor chosen by topology): for a boundary strictly beneath canonical home, open the canonical home directory, require the opened home to be a real directory owned by the EFFECTIVE USER (not root, not another principal) and closed to group/other writes, retain that home fd, and traverse each descendant component relative to the retained fd withO_DIRECTORY|O_NOFOLLOW|O_CLOEXEC, preserving the existing ownership/private-mode checks below home and retaining the verified final directory fd. The walk NEVER opens or inspects/or/home, never uses mount ownership or mountinfo as security evidence, and never globally trusts UID 65534. This home-anchored traversal IS the systemd-compatible security boundary — and it works identically in foreground mode for under-home state directories.run --state-dir <outside-home>keeps the conservative root-to-leaf ownership/permission validation with NO mountinfo/read-only-owner relaxation: an untrusted displayed owner remains untrusted there. Service and foreground contracts are explicit: no mode inference from environment details was needed — install-time policy enforcement (service) and topology-anchored walking (both modes) keep one simple verifier._read_mount_table,_unescape_mount_field,_is_mount_prefix,_mount_point_is_readonly, the_PROC_MOUNTINFOconstant, the sandbox-ancestor exception inside the walk, theMountInfoReadonlyLookupTestsparser suite and theSystemdSandboxNamespaceTestsreadonly matrix. A source-level test (MountinfoIsNotPartOfTheTrustModelTests) pins that the walk module contains no mount-table reference so no relaxation can quietly grow back. The trust module net-shrinks (−16 lines; the entire mount parser gone).O_NOFOLLOW; private final state dir; unit-directory descriptor-bound publication; secure atomic writer; marker ownership protection; POSIX lock bound to retained directory identity; Windows reparse/junction protections; broad//$HOMEwritable-grant rejection; exact path normalization before rendering; no credentials in unit/argv;ProtectSystem=strict;ProtectHome=read-only./and parent view accepted for the under-home boundary — proving those ancestors are outside the trust computation; untrusted owner BELOW home refused without mount reasoning; root-owned final directory below home refused; world-writable directory below home refused; foreign-owned home anchor refused; group-writable home anchor refused). NewServiceHomeContractTests: default state layout installs; explicit private custom state dir beneath home installs; state dir = home refused; state dir =/refused; outside-home state dir refused (paired, path named); symlinked state dir resolving outside home refused;XDG_CONFIG_HOMEoutside home refused with the unit directory not even provisioned;XDG_CONFIG_HOMEbeneath home accepted. Existing attack regressions pass unchanged: planted state database symlink, path replacement, lock leaf symlink, lock directory replacement, unsafe shared paths, unit directory replacement, broad writable grants.systemd-run --usernamespace regression (systemd 255, this host): PASSED —ProtectSystem=strict+ProtectHome=read-only+ReadWritePaths=<private under-home state dir>; the home-anchored walk accepts the legitimate boundary with NO mountinfo exception involved. (Under themake checkrunner the probe remains environment-gated/skipped, as before; executed directly it runs and passes.)w-e69ce40f9c7842e6untouched and NO re-pairing, and NO reinstall (the installed unit is byte-identical to what the new renderer produces for the live flags — verified),systemctl --user start scarcity-router-worker: the service reachesactive (running)and stays there (singleStartedjournal entry,NRestarts=0,ExecMainStatus=0, one stable main PID), ZEROuntrusted worker path componententries, and the worker process holds an ESTABLISHED outbound connection to the loopback DEV Router (127.0.0.1:8790) using the stored identity. The full Phase-A crash/restart/reboot sequence was NOT continued.bfff8a3:uv sync --only-dev,make check(2634 tests OK, 15 platform-gated skips on Linux),uv run basedpyright(0 errors/0 warnings/0 notes),git diff --check.scarcity_router/worker_identity_store.py,scarcity_router/worker_service.py,tests/test_worker_local_store.py,tests/test_worker_service.py,tests/test_worker_client.py,docs/decisions.md,README.md.UX impact: noneon the happy path — the documented default layout (~/.local/share/scarcity-router/worker,~/.config/systemd/user) behaves identically. Rejection paths gained a clearer, actionable boundary: a custom service state dir outside home is now refused at install with the exact path, the supported alternatives and the foreground-runalternative named (necessary security friction; the foreground capability itself is unchanged).Review gates (per the frozen scope — open at the human decision point)
GLM-5.3-Flashreview mandatory; Daybreak security review when available, elseSECURITY_REVIEW_UNAVAILABLEwith the gate open. UX assessment mandatory.UX impact:Sources page text changes (install promoted, run framed as debugging); README worker section rewritten around the service workflow;rununchanged as debugging path. Remediation round 1:UX impact: none(unit rendering and install refusal-path only). Remediation round 2:UX impact: noneon the happy path (relative inputs still accepted and now display their normalized absolute form); rejection paths gained actionable, path-naming errors (broad grants, shared unit directories, planted lock files) — necessary security friction, each with a remediation hint.Scope audit
#139 (TLS/CA) not implemented; #140 admin UX not touched beyond the #138-required Sources hint; #141 not started; #165/#166 untouched; no machine-specific Precision configuration (source ids in docs are the frozen scope's documented examples); no credentials committed; no manual-only hidden dependency (the copied-example path is explicitly retired with a migration message).
Non-goals honored
No Windows service mechanics (M10 tray remains the Windows path); no root system units; no own daemonizing logic; no configuration migration.
Final four-blocker path-identity remediation published as
85b6e714177dc01885415116b7aa9678c2b4d4b4on the existing branch. Reviewed old HEAD:7523bbb07393b1f5020e64ad08fa2cfc32f773ff.origin/develop/PR base remainsdb4e1bde28b2398e94b2911eb57bd3b752538dc5. Remote HEAD verified bygit ls-remoteand Forgejo PR API; PR remains open, targeting develop, unmerged. Worktree clean; no merge commits inorigin/develop..HEAD.Mechanism
Discriminating evidence
Against immutable old HEAD, then remediated tree: renderer workspace preservation false -> true; unsafe DB opener calls 1 -> 0 and victim mode 0600 -> unchanged 0644 (bytes unchanged); unit replacement writes unchecked B/systemctl called -> B untouched, publication retained in A, refusal before systemctl; POSIX lock in unchecked B -> lock only in validated A. Native Windows state/ancestor junction probes: old HEAD acquires and changes victim -> new HEAD refuses with victim unchanged.
Validation
uv sync --only-dev: PASS.C:\Python314\python.exe, WSL interop): 15 lock adversarial tests + 17 tray tests, 32/32 PASS, zero skips. Includes final symlink/junction, state/ancestor junctions, successful in-place FSCTL attack followed by safe refusal, first-run provisioning, contention/PID, cleanup and process-death release. Independent narrow verification separately passed 15/15 native lock tests.make check: 2618 tests OK (15 Windows-only skips covered natively), basedpyright 0 errors/0 warnings/0 notes.uv run basedpyrightandgit diff --check: PASS.UX impact: intentional security narrowing only. Custom state must already exist/private/paired; symlinked unit-directory chains require private canonical XDG_CONFIG_HOME; Windows reparse chains refuse. Default first-run remains automatic, and errors/README explain recovery.
Files: README.md, docs/decisions.md, worker_identity_store.py, worker_local_store.py, worker_service.py, test_worker_local_store.py, test_worker_service.py. No generic filesystem-security framework. CR/LF renderer hardening, marker first-line refinement, O1–O8, #139/#140/#141/#165/#166 and all unrelated observations remain out of scope. No merge performed.
Phase-A live acceptance failed inside the generated unit: under ProtectSystem=strict + ProtectHome=read-only, systemd establishes its read-only mounts in a mount namespace whose user mapping excludes host root, so / and /home legitimately report the overflow uid 65534 and the trusted-ancestor walk refused them ("untrusted worker path component /", exit 2, restart loop). The walk now accepts a NON-FINAL ancestor with an untrusted displayed owner only when its mode is closed to group/other writes AND the mount it resides on is read-only in the process's own /proc/self/mountinfo view: the kernel then denies every principal the rename/unlink the ownership check exists for. UID 65534 is not globally trusted — an untrusted owner on a writable mount is still refused, the boundary itself must still be owned by the invoking user, and all fd-relative O_NOFOLLOW identity checks are unchanged. D-065 records the namespace-aware clause. Regression coverage: deterministic sandbox-view matrix (synthetic ancestors accepted; writable-mount nobody refused; untrusted final dir refused even on a read-only mount; world-writable ancestor refused even in the sandbox view), mountinfo longest-prefix/stacked-mount/escaping parser tests, and a real systemd-run --user acceptance probe (skipped where no user manager exists) that was itself run live on ROG: the sandbox accepts the private under-home boundary, and the service now starts, stays active and connects to the loopback DEV router with the stored identity.