Creatidy/scarcity-router

Fork 0

F. First-class worker service lifecycle (systemd) #138

New issue

Closed

opened 2026-09-26 21:43:47 +00:00 by kilo · 2 comments

kilo commented

2026-09-26 21:43:47 +00:00

Member

Child F of #132. The Precision Codex worker was manually moved from foreground execution to systemd — make that product functionality with a first-class Linux/WSL worker service UX.

Scope (frozen)

scarcity-router-worker service install --codex-source precision-codex-live
scarcity-router-worker service status
scarcity-router-worker service restart
scarcity-router-worker service uninstall

Requirements:

Operates on an already-paired worker identity; uses the stored server origin; preserves selected codex source(s) and any local Ollama configuration.
Generates a systemd user unit from the ACTUAL installed executable and the ACTUAL worker state directory; no API keys or pairing secrets in the unit.
Custom CA configuration, when used, is persisted through the supported worker configuration (child G's --ca-cert persistence) and automatically reused — unit files never need ad-hoc SSL_CERT_FILE shell hacks.
Install is idempotent/atomic; daemon-reload/enable/start correct; status useful; restart/uninstall safe; handles loginctl linger deliberately and visibly; never silently overwrites an unrelated existing unit.
Lifetime single-instance lock per worker state directory: a foreground worker and a systemd worker cannot run concurrently; a second process fails closed with a useful message.
Sources UI promotes service installation for normal deployment; foreground run is framed as debugging/development.

Tests

Unit generation (idempotent, no secrets — negative test scans unit content for credential material), install/restart/uninstall safety against a fake systemctl, overwrite-refusal, lock behavior (second instance fails closed), state-directory correctness, source/Ollama/CA preservation.

Security classification

Security-sensitive (service lifecycle, credential-adjacent surface): independent GLM-5.3-Flash review mandatory; Daybreak security review when available, else SECURITY_REVIEW_UNAVAILABLE with the gate open. UX assessment mandatory.

Complexity boundary

New worker_service.py-class module + worker_client.py subcommand + Sources UI hook + docs (worker/deployment doc in place); ~3 test files. Budget: 120 min wall clock.

Non-goals

No Windows service mechanics (M10 tray remains the Windows path); no root system units; no daemonizing logic of our own (systemd owns the lifecycle); no configuration migration.

Child F of #132. The Precision Codex worker was manually moved from foreground execution to systemd — make that product functionality with a first-class Linux/WSL worker service UX. ## Scope (frozen) ``` scarcity-router-worker service install --codex-source precision-codex-live scarcity-router-worker service status scarcity-router-worker service restart scarcity-router-worker service uninstall ``` Requirements: - Operates on an already-paired worker identity; uses the stored server origin; preserves selected codex source(s) and any local Ollama configuration. - Generates a systemd **user** unit from the ACTUAL installed executable and the ACTUAL worker state directory; no API keys or pairing secrets in the unit. - Custom CA configuration, when used, is persisted through the supported worker configuration (child G's `--ca-cert` persistence) and automatically reused — unit files never need ad-hoc `SSL_CERT_FILE` shell hacks. - Install is idempotent/atomic; daemon-reload/enable/start correct; status useful; restart/uninstall safe; handles `loginctl linger` deliberately and visibly; never silently overwrites an unrelated existing unit. - **Lifetime single-instance lock per worker state directory**: a foreground worker and a systemd worker cannot run concurrently; a second process fails closed with a useful message. - Sources UI promotes service installation for normal deployment; foreground `run` is framed as debugging/development. ## Tests Unit generation (idempotent, no secrets — negative test scans unit content for credential material), install/restart/uninstall safety against a fake systemctl, overwrite-refusal, lock behavior (second instance fails closed), state-directory correctness, source/Ollama/CA preservation. ## Security classification Security-sensitive (service lifecycle, credential-adjacent surface): independent GLM-5.3-Flash review mandatory; Daybreak security review when available, else `SECURITY_REVIEW_UNAVAILABLE` with the gate open. UX assessment mandatory. ## Complexity boundary New `worker_service.py`-class module + `worker_client.py` subcommand + Sources UI hook + docs (worker/deployment doc in place); ~3 test files. Budget: 120 min wall clock. ## Non-goals No Windows service mechanics (M10 tray remains the Windows path); no root system units; no daemonizing logic of our own (systemd owns the lifecycle); no configuration migration.

kilo referenced this issue

2026-09-27 16:21:42 +00:00

Harness-independent execution backend: architecture boundary, semantic harness contract and local-transport policy (D-056, program #132) #146

kilo referenced this issue

2026-09-27 21:01:22 +00:00

Representative-client compatibility + operations hardening program (ZCode acceptance follow-up) #132

kilo commented

2026-09-27 21:01:49 +00:00

Author

Member

Additive scope note (D-056, PR #147 — frozen scope text above unchanged; the scope itself is unaffected):

D-056's local-transport tiers govern CLIENT traffic to the composed HTTP listener only and deliberately do NOT relax worker transport (D-056 pt 8E; the boundary note in docs/worker-protocol.md). For this child: generated service units and service tooling must keep requiring verified-TLS srws:// worker origins in production — worker loopback plaintext remains the bounded dev/tests-only exception, and nothing in the newly accepted local-HTTP tiers may introduce a plaintext worker origin, weakened verification or weakened credential handling here.

Additive scope note (D-056, PR #147 — frozen scope text above unchanged; the scope itself is unaffected): D-056's local-transport tiers govern CLIENT traffic to the composed HTTP listener only and deliberately do NOT relax worker transport (D-056 pt 8E; the boundary note in `docs/worker-protocol.md`). For this child: generated service units and service tooling must keep requiring verified-TLS `srws://` worker origins in production — worker loopback plaintext remains the bounded dev/tests-only exception, and nothing in the newly accepted local-HTTP tiers may introduce a plaintext worker origin, weakened verification or weakened credential handling here.

adrian.tkacz referenced this issue from a commit

2026-10-01 01:48:07 +00:00

First-class worker service lifecycle: generated systemd user unit (D-065, #138)

kilo referenced this issue from a pull request that will close it,

2026-10-01 01:48:36 +00:00

First-class worker service lifecycle: generated systemd user unit (D-065, #138) #167

adrian.tkacz referenced this issue from a commit

2026-10-01 09:44:12 +00:00

Review blockers: specifier-safe unit rendering + exact ZCode state home write grant (#138)

adrian.tkacz referenced this issue from a commit

2026-10-01 12:39:01 +00:00

Daybreak review blockers: canonical persisted paths, writable-grant policy, secure unit write, no-follow lock (#138)

adrian.tkacz referenced this issue from a commit

2026-10-01 13:43:29 +00:00

Bind validated worker paths to retained object identity (#138)

adrian.tkacz referenced this issue from a commit

2026-10-01 21:17:50 +00:00

Accept systemd's synthetic read-only sandbox ancestors in the trusted-path walk (#138)

adrian.tkacz referenced this issue from a commit

2026-10-02 22:33:37 +00:00

Anchor the worker trust walk at the user's home; drop mountinfo (D-065, #138)

kilo commented

2026-10-03 08:46:28 +00:00

Author

Member

Phase-A live acceptance record — ROG (finalized 2026-10-03)

Verdict: ACCEPTANCE_A_PASS_PHASE_B_DEFERRED. Phase B (reboot persistence) is PHASE_B_DEFERRED_BY_OWNER; its absence is not a Phase-A failure. This record consolidates the Phase-A campaign (2026-10-01 → 2026-10-03) and the finalization-state verification at the exact final HEAD. No destructive test was repeated during finalization; the only new live execution was the non-destructive single-instance-lock refusal check.

Environment under test

PR #167 head bfff8a392fae40e021b8d6143cd7ff08ca7f3ef9 (branch task/138-worker-service-lifecycle, base develop @ db4e1bde28b2398e94b2911eb57bd3b752538dc5; PR open, unmerged, mergeable). Local checkout HEAD identical, worktree clean, remote branch head verified equal; origin/develop..HEAD = 6 commits, 0 merge commits.
Host: ROG, WSL2 (kernel 6.18.33.2-microsoft-standard-WSL2), systemd 255.4-1ubuntu8.17, user manager PID 3134755 (systemd --user). Boot ID df051e5a-8897-4fc6-986b-34feac17b0dd (no reboot since 2026-09-28 — Phase B not exercised, by scope).
Router: native local DEV Router (scarcity_router.control_server), PID 3148529, worker protocol listener 127.0.0.1:8790.
Worker identity: w-e69ce40f9c7842e6, paired 2026-10-01T20:18:52Z; identity row untouched since — no re-pairing through the entire campaign. Stored origin srw://127.0.0.1:8790.
Unit: /home/adrian/.config/systemd/user/scarcity-router-worker.service, sha-256 dac3713ba3b53de24501a70800e4caeda8f090ec0240786e8f5d056f6ac2c948, written 2026-10-01 22:20:02 CEST and never rewritten since (mtime unchanged through all tests). enabled; Linger=yes; managed-by marker present; ExecStart → durable repo-venv executable, --state-dir pinned; ProtectSystem=strict; ProtectHome=read-only; ReadWritePaths = worker state dir only; Restart=on-failure / RestartSec=5s; TimeoutStopSec=30s; NoNewPrivileges/PrivateTmp/RestrictAddressFamilies present. No pairing code or credential anywhere in the unit (verified by direct read; test negative-scan also covers it). systemd-analyze verify clean.
Worker state dir: /home/adrian/.local/share/scarcity-router/worker (0700 tree; worker-state.db 0600; worker.lock 0600 — reserved byte 0 + holder-pid line).
Current service state at finalization: active (running), MainPID 4105320 since 2026-10-03 10:33:32 CEST, NRestarts=0, Result=success.
Live connection: ESTABLISHED 127.0.0.1:34088 → 127.0.0.1:8790, client side owned by MainPID 4105320, server side by the DEV Router. Prior worker generations logged session accepted at protocol v4 on connect (D-064 protocol).

Phase-A items

A. Service installation — PASS. Generated + enabled + managed-by marker, no secrets, durable executable, canonical state dir, hardening settings, linger enabled — all re-verified from the live unit at final HEAD (details above).
B. Worker connection — PASS. Live established outbound loopback connection above; worker active (running), stable PID, ~18 MB RSS.
C. Single-instance lock — PASS. Re-verified live at final HEAD: a foreground run --state-dir … against the active service refused fail-closed with exit 2: another worker process is already running for this worker state directory (pid 4105320); stop it first (with the service installed: systemctl --user stop scarcity-router-worker), or choose a different --state-dir. The service was untouched by the attempt (identical MainPID, connection and NRestarts re-verified immediately after); worker.lock shows the holder pid and is held for the service's lifetime. Matches the prior recorded refusal.
D. Clean SIGTERM stop — PASS. Journal 2026-10-02 23:37:52 and 2026-10-03 10:28:37 CEST: Stopping → worker graceful connection lost (deterministic stop path) → Stopped; no Failed with result, no SIGKILL escalation, no automatic restart after either stop. Contrast within the same journal: the 2026-10-01 22:20 failure pattern did log Failed with result 'exit-code'.
E. Lock release after clean stop — PASS. After each clean stop the service started again successfully (2026-10-03 00:31:05 and 10:31:15), acquiring the lock; item F additionally proves reacquisition after SIGKILL.
F. Crash/restart — PASS. Journal 2026-10-03 10:32:11 CEST: Main process exited, code=killed, status=9/KILL + Failed with result 'signal'; 10:32:16: Scheduled restart job, restart counter is at 1 → Started; the restarted worker logged session accepted at protocol v4 (reconnected to 127.0.0.1:8790); identity unchanged — no re-pairing.
G. Product service restart — PASS. Journal 2026-10-03 10:33:32 CEST: Stopping → Stopped → Started within ~54 ms (systemctl --user restart driven by the product CLI; prior recorded CLI output: worker service: restarted scarcity-router-worker.service). Current MainPID 4105320 dates from that restart and holds the live connection.
H. Idempotent service install — PASS. Prior recorded no-op re-install (identical PID / unit SHA-256 / mtime; service stayed active) corroborated at finalization: unit mtime still 2026-10-01 22:20:02, on-disk unit byte-identical to a fresh render_unit output at HEAD bfff8a3 (deterministic renderer), systemd-analyze verify clean.

Notes: journal timestamps above are local CEST; the worker's own log lines are UTC (Z). The installed unit's ExecStart is run --state-dir … with no adapter-selection flags — install preserved this deployment's (empty) selection; flag preservation across selections is covered by unit tests. The 2026-10-01 22:20 restart loop in the journal is the historical round-3 synthetic-namespace bug (untrusted worker path component /), fixed in rounds 3–4 before final HEAD; it has not recurred since 2026-10-01 23:14 (24 h+ stable run to the first clean stop).

Credential / logging observations

No secret appears in the unit, the journal, CLI output or error messages. The pairing credential lives only in the 0700 state tree (0600 files); inspection during finalization was read-only (SQLite mode=ro) and selected only non-secret fields (worker id, origin, label). Journal worker lines are limited to session/connection events.

UX verdict: `UX_PASS`

The administrator workflow pair → service install → service status → systemd lifecycle → restart/recovery → idempotent re-install was operable end-to-end: no manual editing of the unit was ever required, and no credential was ever copied into a persistent command or config (the pairing code is one-time; the identity lives in the 0700/0600 state store; the unit embeds none). Linger is handled and its outcome reported by install (Linger=yes). Observed friction, all resolved or non-blocking: (1) the initial systemd synthetic-namespace bug was discovered live via an actionable error and fixed before final HEAD; (2) the user-manager/linger prerequisite on WSL required the documented prerequisite step; (3) refusal/error messages name the holder pid and the exact remediation command (observed live in C). Minor wording/polish goes to #140.

Phase B

PHASE_B_DEFERRED_BY_OWNER — automatic service return after WSL/host restart (no interactive login, no re-pairing) is deliberately not exercised in this record; the host has not rebooted since 2026-09-28. This is not a Phase-A failure.

Remaining non-blocking follow-ups

#139 (TLS/CA persistence), #140 (admin UX polish), #141, #165, #166 — untouched, per frozen scope.
Daybreak/security review gate status unchanged per the PR #167 body; owner review and the human merge gate remain open. Nothing is closed or merged by this record.

# Phase-A live acceptance record — ROG (finalized 2026-10-03) Verdict: **ACCEPTANCE_A_PASS_PHASE_B_DEFERRED**. Phase B (reboot persistence) is `PHASE_B_DEFERRED_BY_OWNER`; its absence is not a Phase-A failure. This record consolidates the Phase-A campaign (2026-10-01 → 2026-10-03) and the finalization-state verification at the exact final HEAD. No destructive test was repeated during finalization; the only new live execution was the non-destructive single-instance-lock refusal check. ## Environment under test - PR #167 head `bfff8a392fae40e021b8d6143cd7ff08ca7f3ef9` (branch `task/138-worker-service-lifecycle`, base `develop` @ `db4e1bde28b2398e94b2911eb57bd3b752538dc5`; PR open, unmerged, mergeable). Local checkout HEAD identical, worktree clean, remote branch head verified equal; `origin/develop..HEAD` = 6 commits, 0 merge commits. - Host: ROG, WSL2 (kernel 6.18.33.2-microsoft-standard-WSL2), systemd 255.4-1ubuntu8.17, user manager PID 3134755 (`systemd --user`). Boot ID `df051e5a-8897-4fc6-986b-34feac17b0dd` (no reboot since 2026-09-28 — Phase B not exercised, by scope). - Router: native local DEV Router (`scarcity_router.control_server`), PID 3148529, worker protocol listener `127.0.0.1:8790`. - Worker identity: `w-e69ce40f9c7842e6`, paired 2026-10-01T20:18:52Z; identity row untouched since — **no re-pairing** through the entire campaign. Stored origin `srw://127.0.0.1:8790`. - Unit: `/home/adrian/.config/systemd/user/scarcity-router-worker.service`, sha-256 `dac3713ba3b53de24501a70800e4caeda8f090ec0240786e8f5d056f6ac2c948`, written 2026-10-01 22:20:02 CEST and never rewritten since (mtime unchanged through all tests). `enabled`; `Linger=yes`; managed-by marker present; `ExecStart` → durable repo-venv executable, `--state-dir` pinned; `ProtectSystem=strict`; `ProtectHome=read-only`; `ReadWritePaths` = worker state dir only; `Restart=on-failure` / `RestartSec=5s`; `TimeoutStopSec=30s`; `NoNewPrivileges`/`PrivateTmp`/`RestrictAddressFamilies` present. **No pairing code or credential anywhere in the unit** (verified by direct read; test negative-scan also covers it). `systemd-analyze verify` clean. - Worker state dir: `/home/adrian/.local/share/scarcity-router/worker` (0700 tree; `worker-state.db` 0600; `worker.lock` 0600 — reserved byte 0 + holder-pid line). - Current service state at finalization: `active (running)`, MainPID 4105320 since 2026-10-03 10:33:32 CEST, `NRestarts=0`, `Result=success`. - Live connection: **ESTABLISHED** `127.0.0.1:34088 → 127.0.0.1:8790`, client side owned by MainPID 4105320, server side by the DEV Router. Prior worker generations logged `session accepted at protocol v4` on connect (D-064 protocol). ## Phase-A items - **A. Service installation — PASS.** Generated + enabled + managed-by marker, no secrets, durable executable, canonical state dir, hardening settings, linger enabled — all re-verified from the live unit at final HEAD (details above). - **B. Worker connection — PASS.** Live established outbound loopback connection above; worker `active (running)`, stable PID, ~18 MB RSS. - **C. Single-instance lock — PASS.** Re-verified live at final HEAD: a foreground `run --state-dir …` against the active service refused fail-closed with exit 2: `another worker process is already running for this worker state directory (pid 4105320); stop it first (with the service installed: systemctl --user stop scarcity-router-worker), or choose a different --state-dir`. The service was untouched by the attempt (identical MainPID, connection and `NRestarts` re-verified immediately after); `worker.lock` shows the holder pid and is held for the service's lifetime. Matches the prior recorded refusal. - **D. Clean SIGTERM stop — PASS.** Journal 2026-10-02 23:37:52 and 2026-10-03 10:28:37 CEST: `Stopping` → worker graceful `connection lost` (deterministic stop path) → `Stopped`; no `Failed with result`, no SIGKILL escalation, no automatic restart after either stop. Contrast within the same journal: the 2026-10-01 22:20 failure pattern did log `Failed with result 'exit-code'`. - **E. Lock release after clean stop — PASS.** After each clean stop the service started again successfully (2026-10-03 00:31:05 and 10:31:15), acquiring the lock; item F additionally proves reacquisition after SIGKILL. - **F. Crash/restart — PASS.** Journal 2026-10-03 10:32:11 CEST: `Main process exited, code=killed, status=9/KILL` + `Failed with result 'signal'`; 10:32:16: `Scheduled restart job, restart counter is at 1` → `Started`; the restarted worker logged `session accepted at protocol v4` (reconnected to 127.0.0.1:8790); identity unchanged — no re-pairing. - **G. Product `service restart` — PASS.** Journal 2026-10-03 10:33:32 CEST: `Stopping` → `Stopped` → `Started` within ~54 ms (`systemctl --user restart` driven by the product CLI; prior recorded CLI output: `worker service: restarted scarcity-router-worker.service`). Current MainPID 4105320 dates from that restart and holds the live connection. - **H. Idempotent `service install` — PASS.** Prior recorded no-op re-install (identical PID / unit SHA-256 / mtime; service stayed active) corroborated at finalization: unit mtime still 2026-10-01 22:20:02, on-disk unit **byte-identical to a fresh `render_unit` output at HEAD `bfff8a3`** (deterministic renderer), `systemd-analyze verify` clean. Notes: journal timestamps above are local CEST; the worker's own log lines are UTC (`Z`). The installed unit's `ExecStart` is `run --state-dir …` with no adapter-selection flags — install preserved this deployment's (empty) selection; flag preservation across selections is covered by unit tests. The 2026-10-01 22:20 restart loop in the journal is the historical round-3 synthetic-namespace bug (`untrusted worker path component /`), fixed in rounds 3–4 before final HEAD; it has not recurred since 2026-10-01 23:14 (24 h+ stable run to the first clean stop). ## Credential / logging observations No secret appears in the unit, the journal, CLI output or error messages. The pairing credential lives only in the 0700 state tree (0600 files); inspection during finalization was read-only (SQLite `mode=ro`) and selected only non-secret fields (worker id, origin, label). Journal worker lines are limited to session/connection events. ## UX verdict: `UX_PASS` The administrator workflow pair → service install → service status → systemd lifecycle → restart/recovery → idempotent re-install was operable end-to-end: no manual editing of the unit was ever required, and no credential was ever copied into a persistent command or config (the pairing code is one-time; the identity lives in the 0700/0600 state store; the unit embeds none). Linger is handled and its outcome reported by install (`Linger=yes`). Observed friction, all resolved or non-blocking: (1) the initial systemd synthetic-namespace bug was discovered live via an actionable error and fixed before final HEAD; (2) the user-manager/linger prerequisite on WSL required the documented prerequisite step; (3) refusal/error messages name the holder pid and the exact remediation command (observed live in C). Minor wording/polish goes to #140. ## Phase B `PHASE_B_DEFERRED_BY_OWNER` — automatic service return after WSL/host restart (no interactive login, no re-pairing) is deliberately not exercised in this record; the host has not rebooted since 2026-09-28. This is not a Phase-A failure. ## Remaining non-blocking follow-ups - #139 (TLS/CA persistence), #140 (admin UX polish), #141, #165, #166 — untouched, per frozen scope. - Daybreak/security review gate status unchanged per the PR #167 body; owner review and the human merge gate remain open. Nothing is closed or merged by this record.

adrian.tkacz closed this issue

2026-10-03 08:58:19 +00:00

adrian.tkacz referenced this issue from a commit

2026-10-03 08:58:20 +00:00

Merge pull request 'First-class worker service lifecycle: generated systemd user unit (D-065, #138)' (#167) from task/138-worker-service-lifecycle into develop