G. Integration + live acceptance + timing report #123

New issue

Closed

opened 2026-09-24 15:23:51 +00:00 by kilo · 1 comment

kilo commented

2026-09-24 15:23:51 +00:00

Member

Child G of #116 (Dynamic Execution Sources program). Integration + acceptance.

Testing (discriminating, real protocol-shaped fixtures; no behavior resting solely on mocked string matching)

explicit {} params on Codex RPC; inventory parsing + bounds; source authentication states; dynamic model appearance; model disappearance; unknown model classification; track classification; automatic safe adoption; exact execution binding after dynamic selection; no silent target substitution; multiple Codex sources on one worker; strict credential/home isolation; same physical model from two accounts stays two resources; quota pools not accidentally merged; source failure isolation; restart/persistence; worker/server ownership; Daybreak restricted-access behavior; required security-review gate; label-first UX; provider-auth vs worker-auth diagnostics; upgrade without manual model reconfiguration.

Test timing observability

Use/extend the existing SR_TEST_TIMING=1 runner (tools/test_runner.py). Final report MUST include top 10 slowest tests with elapsed times and what makes them expensive. No bureaucratic gate-class splits for timing; long tests acceptable when justified.

Upgrade acceptance (central criterion)

T0: source exposes gpt-6-sol. T1: provider adds gpt-6.1-sol. Expected: discover → classify openai/sol → apply conservative track policy → routable when gates pass; the user does NOT edit server configuration; existing in-flight/exact pins unchanged. Also simulate disappearance and unknown-family discovery.

Gates

make check, make package-check, make guardrails, git diff --check, focused suites. No warning suppression; no flaky retry.

Live acceptance (real Codex, product path only)

Existing controlled live Codex source on Precision (codex-cli 0.155.0-alpha.16.3); re-login only if necessary. Product path must automatically discover current gpt-6-luna/sol/astra with runtime-reported efforts; no manual model resource creation. Then: source authenticated → inventory visible → exact resources materialized → selected resource eligible → (issue one inference client key if needed) → ONE real request through the normal OpenAI-compatible endpoint → router selects an appropriate GPT-6 target → record exact source/model/effort → verify full path client → server → worker → controlled Codex source → exact GPT-6 model → response. No direct adapter invocation counts as completion.

Daybreak posture: show separately as restricted; verify actual access before declaring usable; record requested alias + resolved model where available; use only for the mandatory security review when authorized.

Multi-account acceptance: deterministically prove two isolated Codex sources in one worker process; offer live two-account acceptance if the owner has a second real account; ownership of a second account is NOT a blocker for deterministic acceptance.

UX acceptance: explicit end-user walkthrough — normal Codex setup requires NO exact model slug, resource-ID construction, adapter-ID knowledge, CODEX_HOME path knowledge, or config edits when GPT-6.x changes. Flow feels like: Add Codex source → label → login → connected → models discovered automatically. Web UI shows friendly labels before opaque IDs; errors actionable; reviewer actively hunts unnecessary manual steps.

Release policy

Do NOT resume v0.1.0 because the program is green. Report release readiness, remaining external gates, known UX debt, known security debt; stop at the human decision. No tag, no PyPI, no GitHub Release, no merge to main.

Child G of #116 (Dynamic Execution Sources program). Integration + acceptance. ## Testing (discriminating, real protocol-shaped fixtures; no behavior resting solely on mocked string matching) explicit `{}` params on Codex RPC; inventory parsing + bounds; source authentication states; dynamic model appearance; model disappearance; unknown model classification; track classification; automatic safe adoption; exact execution binding after dynamic selection; no silent target substitution; multiple Codex sources on one worker; strict credential/home isolation; same physical model from two accounts stays two resources; quota pools not accidentally merged; source failure isolation; restart/persistence; worker/server ownership; Daybreak restricted-access behavior; required security-review gate; label-first UX; provider-auth vs worker-auth diagnostics; upgrade without manual model reconfiguration. ## Test timing observability Use/extend the existing `SR_TEST_TIMING=1` runner (tools/test_runner.py). Final report MUST include top 10 slowest tests with elapsed times and what makes them expensive. No bureaucratic gate-class splits for timing; long tests acceptable when justified. ## Upgrade acceptance (central criterion) T0: source exposes `gpt-6-sol`. T1: provider adds `gpt-6.1-sol`. Expected: discover → classify `openai/sol` → apply conservative track policy → routable when gates pass; the user does NOT edit server configuration; existing in-flight/exact pins unchanged. Also simulate disappearance and unknown-family discovery. ## Gates `make check`, `make package-check`, `make guardrails`, `git diff --check`, focused suites. No warning suppression; no flaky retry. ## Live acceptance (real Codex, product path only) Existing controlled live Codex source on Precision (`codex-cli 0.155.0-alpha.16.3`); re-login only if necessary. Product path must automatically discover current `gpt-6-luna/sol/astra` with runtime-reported efforts; no manual model resource creation. Then: source authenticated → inventory visible → exact resources materialized → selected resource eligible → (issue one inference client key if needed) → ONE real request through the normal OpenAI-compatible endpoint → router selects an appropriate GPT-6 target → record exact source/model/effort → verify full path `client → server → worker → controlled Codex source → exact GPT-6 model → response`. No direct adapter invocation counts as completion. Daybreak posture: show separately as restricted; verify actual access before declaring usable; record requested alias + resolved model where available; use only for the mandatory security review when authorized. Multi-account acceptance: deterministically prove two isolated Codex sources in one worker process; offer live two-account acceptance if the owner has a second real account; ownership of a second account is NOT a blocker for deterministic acceptance. UX acceptance: explicit end-user walkthrough — normal Codex setup requires NO exact model slug, resource-ID construction, adapter-ID knowledge, CODEX_HOME path knowledge, or config edits when GPT-6.x changes. Flow feels like: Add Codex source → label → login → connected → models discovered automatically. Web UI shows friendly labels before opaque IDs; errors actionable; reviewer actively hunts unnecessary manual steps. ## Release policy Do NOT resume v0.1.0 because the program is green. Report release readiness, remaining external gates, known UX debt, known security debt; stop at the human decision. No tag, no PyPI, no GitHub Release, no merge to main.

kilo referenced this issue

2026-09-24 19:14:20 +00:00

Dynamic Execution Sources, Model Discovery, Multi-Account & Security #116

kilo commented

2026-09-26 17:55:12 +00:00

Author

Member

Closeout — implemented in merged PR #124 (merge commit a4370f0 into develop, head 9789c67; program #116 / D-053).

Child G delivered: the full discriminating suite (2,044 tests at closeout; SR_TEST_TIMING observability retained) covering every listed behavior — explicit {} params, inventory bounds, per-effort resources, dynamic appear/disappear/unknown, exact execution binding, multi-source isolation, quota-pool separation, Daybreak restricted-access, auth-honesty diagnostics, and the upgrade acceptance (T0→T1 with zero configuration changes, verified e2e and live). Final gates at 9789c67: make check (2,044 tests OK), make package-check, make guardrails, git diff --check — all green; top slowest tests are the real-TLS multi-process e2e scenarios (~6–12 s each), documented in the program report.

Live acceptance PASSED through the product path on Precision's real authenticated source: discovery (authenticated, GPT-6 Luna/Sol/Astra materialized per-effort) → one real request through /v1/chat/completions pinned personal-openai:gpt-6-sol:high → 200 with LIVE-ACCEPTANCE-OK, real provider usage, audit executed-target == pin. Multi-account deterministically proven (two sources, one worker, isolated homes/pools). No tag, no PyPI, no merge to main — release posture reported to the owner as required.

**Closeout — implemented in merged PR #124** (merge commit `a4370f0` into `develop`, head `9789c67`; program #116 / D-053). Child G delivered: the full discriminating suite (2,044 tests at closeout; SR_TEST_TIMING observability retained) covering every listed behavior — explicit `{}` params, inventory bounds, per-effort resources, dynamic appear/disappear/unknown, exact execution binding, multi-source isolation, quota-pool separation, Daybreak restricted-access, auth-honesty diagnostics, and the upgrade acceptance (T0→T1 with zero configuration changes, verified e2e and live). Final gates at `9789c67`: `make check` (2,044 tests OK), `make package-check`, `make guardrails`, `git diff --check` — all green; top slowest tests are the real-TLS multi-process e2e scenarios (~6–12 s each), documented in the program report. Live acceptance PASSED through the product path on Precision's real authenticated source: discovery (`authenticated`, GPT-6 Luna/Sol/Astra materialized per-effort) → one real request through `/v1/chat/completions` pinned `personal-openai:gpt-6-sol:high` → 200 with `LIVE-ACCEPTANCE-OK`, real provider usage, audit executed-target == pin. Multi-account deterministically proven (two sources, one worker, isolated homes/pools). No tag, no PyPI, no merge to main — release posture reported to the owner as required.

kilo closed this issue

2026-09-26 17:55:33 +00:00