Creatidy/scarcity-router

Fork 0

fix(telemetry): classify evidenced Codex auth failure as auth_required / telemetry_auth_required #101

New issue

Closed

opened 2026-09-19 23:17:39 +00:00 by kilo · 1 comment

kilo commented

2026-09-19 23:17:39 +00:00

Member

Goal

Make the Router's OpenAI telemetry distinguish an authentication-required condition (expired/unrecoverable persisted ChatGPT credential) from generic telemetry/schema failure, so operators see the bounded reason telemetry_auth_required instead of the ambiguous telemetry_invalid. Routing must remain fail-closed.

Confirmed incident evidence (Precision PRD, 2026-09-19)

During the Codex ChatGPT credential-expiry incident (Infrastructure/creatidy-autonomy#50), the Router reported state=unknown, reason_codes=["telemetry_invalid"] while the actual upstream condition was authentication: app-server account/rateLimits/read → JSON-RPC -32603 → backend 401 Unauthorized → token expired. That diagnosis cost a manual production investigation; the code already contains every vocabulary element needed for the precise answer.

Why the reason code is currently unreachable (verified on develop `fce4311`)

_STATUS_VALUES already contains snapshot status auth_required, _DIAGNOSTIC_CODES contains diagnostic auth_required, and ELIGIBILITY_REASON_CODES contains telemetry_auth_required (capacity.py, eligibility.py).
_failure_eligibility() already maps status auth_required → reason telemetry_auth_required (providers/openai_codex_acquisition.py).
But _run_session() never produces the auth_required status: after the evidenced -32603 trigger, the bounded D-018 recovery failure paths (refresh response carries a validated protocol error; retry response carries a validated protocol error) return _observation("unknown", "telemetry_unknown"), which normalizes to telemetry_invalid.

Scope

In _run_session(): when the rate-limits read fails with the evidenced -32603 code and the bounded recovery sequence then fails (refresh response carries a validated protocol error, or the retry carries a validated protocol error), classify the observation as snapshot status/diagnostic auth_required → eligibility unknown with reason telemetry_auth_required. Any other protocol-error code keeps the existing unknown/telemetry_unknown behavior. The initial -32603 alone (before recovery is attempted) must not classify anything by itself.
Update/add unit tests covering: refresh-error → auth_required; retry-error → auth_required; non-−32603 errors unchanged; parse-path telemetry_invalid unchanged.
Docs touch-up where the failure-state mapping is documented (docs/capacity-model.md / docs/decisions.md D-018 wording).

Security invariants (unchanged)

No error bodies, no free-text messages, no tokens, no auth-file contents in any snapshot, reason, log or exception — classification reads only the validated integer protocol-error code, exactly as today.
Fail-closed preserved: auth_required maps to eligibility state unknown; OpenAI stays excluded until a validated rate-limits observation succeeds.
The Router stays a read-only viewer of the credential; refresh ownership and the writable mount remain with the autonomy worker.

Non-goals

No writable auth mount for the Router; no auth-mode classification in the eligibility parser; no retry/loop changes to the bounded D-018 sequence; no REST/MCP surface changes (envelope already carries reason_codes).

Acceptance criteria

Expired-credential condition surfaces as state=unknown, reason_codes=["telemetry_auth_required"] on /v1/status and in the select exclusion evidence.
All existing tests pass; new tests pin the classification boundaries above.
make check green.

Links

Primary incident issue: Infrastructure/creatidy-autonomy#50 (worker-owned credential maintenance).

## Goal Make the Router's OpenAI telemetry distinguish an authentication-required condition (expired/unrecoverable persisted ChatGPT credential) from generic telemetry/schema failure, so operators see the bounded reason `telemetry_auth_required` instead of the ambiguous `telemetry_invalid`. Routing must remain fail-closed. ## Confirmed incident evidence (Precision PRD, 2026-09-19) During the Codex ChatGPT credential-expiry incident (Infrastructure/creatidy-autonomy#50), the Router reported `state=unknown, reason_codes=["telemetry_invalid"]` while the actual upstream condition was authentication: app-server `account/rateLimits/read` → JSON-RPC `-32603` → backend `401 Unauthorized` → token expired. That diagnosis cost a manual production investigation; the code already contains every vocabulary element needed for the precise answer. ## Why the reason code is currently unreachable (verified on develop `fce4311`) - `_STATUS_VALUES` already contains snapshot status `auth_required`, `_DIAGNOSTIC_CODES` contains diagnostic `auth_required`, and `ELIGIBILITY_REASON_CODES` contains `telemetry_auth_required` (`capacity.py`, `eligibility.py`). - `_failure_eligibility()` already maps status `auth_required` → reason `telemetry_auth_required` (`providers/openai_codex_acquisition.py`). - But `_run_session()` never produces the `auth_required` status: after the evidenced `-32603` trigger, the bounded D-018 recovery failure paths (refresh response carries a validated protocol error; retry response carries a validated protocol error) return `_observation("unknown", "telemetry_unknown")`, which normalizes to `telemetry_invalid`. ## Scope - In `_run_session()`: when the rate-limits read fails with the evidenced `-32603` code and the bounded recovery sequence then fails (refresh response carries a validated protocol error, or the retry carries a validated protocol error), classify the observation as snapshot status/diagnostic `auth_required` → eligibility `unknown` with reason `telemetry_auth_required`. Any other protocol-error code keeps the existing `unknown`/`telemetry_unknown` behavior. The initial `-32603` alone (before recovery is attempted) must not classify anything by itself. - Update/add unit tests covering: refresh-error → `auth_required`; retry-error → `auth_required`; non-−32603 errors unchanged; parse-path `telemetry_invalid` unchanged. - Docs touch-up where the failure-state mapping is documented (`docs/capacity-model.md` / `docs/decisions.md` D-018 wording). ## Security invariants (unchanged) - No error bodies, no free-text messages, no tokens, no auth-file contents in any snapshot, reason, log or exception — classification reads only the validated integer protocol-error code, exactly as today. - Fail-closed preserved: `auth_required` maps to eligibility state `unknown`; OpenAI stays excluded until a validated rate-limits observation succeeds. - The Router stays a read-only viewer of the credential; refresh ownership and the writable mount remain with the autonomy worker. ## Non-goals - No writable auth mount for the Router; no auth-mode classification in the eligibility parser; no retry/loop changes to the bounded D-018 sequence; no REST/MCP surface changes (envelope already carries `reason_codes`). ## Acceptance criteria - [ ] Expired-credential condition surfaces as `state=unknown`, `reason_codes=["telemetry_auth_required"]` on `/v1/status` and in the select exclusion evidence. - [ ] All existing tests pass; new tests pin the classification boundaries above. - [ ] `make check` green. ## Links - Primary incident issue: Infrastructure/creatidy-autonomy#50 (worker-owned credential maintenance).

kilo added the

bug

label

2026-09-19 23:18:43 +00:00

adrian.tkacz referenced this issue from a commit

2026-09-20 00:00:28 +00:00

fix(telemetry): classify evidenced Codex auth failure as auth_required (issue #101)

kilo referenced this issue from a pull request that will close it,

2026-09-20 00:00:49 +00:00

fix(telemetry): keep -32603 recovery failure generically fail-closed; document auth-required unreachability (#101) #102

adrian.tkacz referenced this issue from a commit

2026-09-20 00:28:25 +00:00

fix(telemetry): keep -32603 recovery failure generically fail-closed (issue #101 audit)

kilo commented

2026-09-20 00:28:48 +00:00

Author

Member

Owner-review outcome (correction pushed to PR #102, commit 2293c89): the telemetry_auth_required classification is reverted — accuracy over vocabulary reachability.

Structured-signal audit of pinned codex 0.154.0 (source: `codex-rs/app-server`)

No typed error payload exists. error_code.rs constructs every JSONRPCErrorError with data: None — the only members are the integer code and the free-text message (which the router's security contract forbids parsing).
-32603 conflates all backend failures. account_processor.rs (get_account_rate_limits_response): response.map_err(|err| internal_error(format!("failed to fetch codex rate limits: {err}"))) — an expired-token 401, a 403, a 5xx, a timeout or an empty snapshot all collapse into the same -32603. A failed refresh/retry therefore does not prove an authentication condition; a backend outage that outlives one refresh+retry would have been mislabeled auth_required and sent an operator into a pointless device-auth re-login.
The refresh surface carries no signal at all. get_account_response calls refresh_token_if_requested(do_refresh).await and discards the outcome (fire-and-forget); refresh failure never produces an error envelope. This matches the live probe evidence from 2026-09-20: account/read {refreshToken:true} returned a RESULT even while the retry afterwards still failed with -32603.

The only code-level auth-adjacent signal on this surface is -32600 invalid_request ("…authentication required to read rate limits") for a missing/unusable local auth store — but -32600 is also used for unrelated invalid-request conditions, it is not the expired-token shape (production evidenced -32603), and D-018 deliberately never refreshes on it.

Resulting behavior

Failed refresh or failed retry → unknown/telemetry_unknown → eligibility unknown/telemetry_invalid (the pre-#102 fail-closed behavior). Routing is fail-closed either way; only the diagnosis changed back.
telemetry_auth_required stays deliberately unreachable until a future codex generation exposes a bounded, structured, non-secret auth-failure signal (typed error data or a dedicated error code). D-018 now records this amendment and those conditions explicitly.
Operators retain the precise auth diagnosis where it belongs: the worker-side maintenance surface (autonomy-openai-maintain.prd, exit 4 = auth_required) and the primary issue's runbook.

Owner-review outcome (correction pushed to PR #102, commit 2293c89): the `telemetry_auth_required` classification is **reverted** — accuracy over vocabulary reachability. ## Structured-signal audit of pinned codex 0.154.0 (source: `codex-rs/app-server`) 1. **No typed error payload exists.** `error_code.rs` constructs every `JSONRPCErrorError` with `data: None` — the only members are the integer `code` and the free-text `message` (which the router's security contract forbids parsing). 2. **`-32603` conflates all backend failures.** `account_processor.rs` (`get_account_rate_limits_response`): `response.map_err(|err| internal_error(format!("failed to fetch codex rate limits: {err}")))` — an expired-token 401, a 403, a 5xx, a timeout or an empty snapshot all collapse into the same `-32603`. A failed refresh/retry therefore does not prove an authentication condition; a backend outage that outlives one refresh+retry would have been mislabeled `auth_required` and sent an operator into a pointless device-auth re-login. 3. **The refresh surface carries no signal at all.** `get_account_response` calls `refresh_token_if_requested(do_refresh).await` and discards the outcome (fire-and-forget); refresh failure never produces an error envelope. This matches the live probe evidence from 2026-09-20: `account/read {refreshToken:true}` returned a RESULT even while the retry afterwards still failed with `-32603`. The only code-level auth-adjacent signal on this surface is `-32600 invalid_request` ("…authentication required to read rate limits") for a missing/unusable *local* auth store — but `-32600` is also used for unrelated invalid-request conditions, it is not the expired-token shape (production evidenced `-32603`), and D-018 deliberately never refreshes on it. ## Resulting behavior - Failed refresh or failed retry → `unknown`/`telemetry_unknown` → eligibility `unknown`/`telemetry_invalid` (the pre-#102 fail-closed behavior). Routing is fail-closed either way; only the diagnosis changed back. - `telemetry_auth_required` stays deliberately unreachable until a future codex generation exposes a bounded, structured, non-secret auth-failure signal (typed error `data` or a dedicated error code). D-018 now records this amendment and those conditions explicitly. - Operators retain the precise auth diagnosis where it belongs: the worker-side maintenance surface (`autonomy-openai-maintain.prd`, exit 4 = `auth_required`) and the primary issue's runbook.

adrian.tkacz closed this issue

2026-09-20 00:45:53 +00:00

adrian.tkacz referenced this issue from a commit

2026-09-20 00:45:53 +00:00

Merge pull request 'fix(telemetry): keep -32603 recovery failure generically fail-closed; document auth-required unreachability (#101)' (#102) from fix/telemetry-auth-required into develop

kilo referenced this issue

2026-09-21 17:07:05 +00:00

[Auth] Expose OpenAI subscription execution eligibility to selection (M4.1) #82