M1: Implement secure Z.ai quota acquisition

creatidy commented

2026-09-03 18:48:44 +00:00

(Migrated from github.com)

Closes #10

What this adds

The production acquisition boundary for the Z.ai Coding Plan collector: the
existing Kilo credential -> strict credential discovery -> fixed HTTPS quota
GET -> bounded strict JSON decoding -> existing parse_zai_quota_response(...)
-> CapacitySnapshot. The pure parser in scarcity_router/providers/zai.py
is untouched; acquisition adds zero quota semantics of its own.

Acquisition API

collect_zai_capacity(
    *,
    retrieved_at: str,               # caller-supplied; no clock/freshness policy (U-003 untouched)
    auth_file: Path | None = None,   # default: ~/.local/share/kilo/auth.json (evidenced Kilo location)
) -> CapacitySnapshot

The explicit auth_file exists for deterministic tests and controlled local
configuration. The credential path and value never enter the snapshot.

Credential-source contract

Supported layout: a JSON object keyed by provider ID; only the exact
zai-coding-plan key is selected. The key value is inspected only after
the target entry is established; unrelated provider entries are never read
as candidates and never appear in output.
Evidenced shape required: type == "api" and a non-empty string key.
No alternative fields (token, value, apiKey, api_key,
tokenValue) and no token-like types are accepted.
The stored value is sent as-is: no Bearer prefix, no strip, no reformat.
Bounded read: 1 MiB (MAX_AUTH_FILE_BYTES); oversized, unreadable,
missing, invalid-UTF-8 and invalid-JSON files all fail safely to
auth_required. The filesystem path is never included in diagnostics.

Credential header safety (security-review hardening)

A credential is accepted only if it can be represented byte-for-value as
an HTTP header value: a conservative printable-ASCII allowlist (space
through tilde). CR, LF, CRLF, NUL, every other C0 control, DEL, C1 controls
and values the stdlib latin-1 header encoding cannot represent are rejected
during discovery - before any request object or transport exists - so no
malformed credential can ever be formatted into a transport exception
(http.client raises ValueError("Illegal header value %r") /
UnicodeEncodeError containing the value). Accepted values (including
leading/trailing spaces) are still transmitted exactly as stored; nothing is
stripped, normalized or escaped. Unsafe credentials normalize to
auth_required with zero transport calls.

Duplicate-key ambiguity (security-review hardening)

The auth document is parsed with a duplicate-rejecting JSON
object_pairs_hook: any duplicate object key anywhere in the document
fails closed to auth_required with zero requests - a repeated top-level
zai-coding-plan entry, repeated type/key fields inside the target
entry, and (documented conservative choice) duplicates in unrelated entries.
Credential selection never trusts last-key-wins parser overwrite behavior,
and neither candidate value can reach the transport or any output.

Fixed endpoint and destination validation

GET https://api.z.ai/api/monitor/usage/quota/limit is a module constant;
the public API exposes no endpoint override.
is_approved_destination (small, independently testable) rejects at
minimum: non-HTTPS, wrong host, host-suffix tricks
(api.z.ai.example.com), userinfo, explicit unexpected ports, query and
fragments, wrong paths; malformed URLs fail closed.
Validation runs before the Authorization header is attached; if the fixed
constant itself ever failed policy, a RuntimeError is raised before any
header exists (programmer error, not disguised telemetry) - asserted by
test that Authorization is never attached in that case.

Redirect policy

HTTPRedirectHandler.redirect_request is overridden to decline every 3xx
(public NoRedirect handler; open_response builds the opener with it).
Coverage of the real production boundary, network-free:

NoRedirect.redirect_request returns None for 301/302/303/307/308 and
constructs no replacement request;
open_response obtains its opener via urllib.request.build_opener with
exactly the NoRedirect handler and passes the original authenticated
request object exactly once - the test fails if production ever switches
to urlopen(...) or a default redirect-following opener.

Mapping tests additionally assert per 3xx code: redirect not followed, zero
second requests, credential never forwarded, snapshot contains no raw
Location.

Authorization behavior and bounds

Headers: Authorization: <stored credential exactly> and
Accept: application/json; no other secret-bearing header.
Exactly one request, one credential use, no retries, no Bearer fallback.
Finite timeout: TIMEOUT_SECONDS = 15.
Response body bounded at 64 KiB (MAX_BODY_BYTES); oversized bodies are
neither parsed nor persisted. Auth-file read bounded at 1 MiB.

Failure/status mapping (v1 capacity contract)

Condition	status	diagnostic
Credential missing/unreadable/malformed/wrong shape/unsafe header value/ambiguous	`auth_required`	`auth_required`
HTTP 401	`auth_required`	`auth_required`
Success response with malformed UTF-8/JSON/empty body	`schema_changed`	`schema_changed`
Connection/DNS/timeout/malformed HTTP protocol/mid-body failure	`unavailable`	`source_unavailable`
Redirect rejected, oversized body, other unevidenced HTTP errors (403/429/5xx...)	`unknown`	`telemetry_unknown`

No raw transport exception escapes; invalid retrieved_at still fails
through the typed capacity-contract validation instead of being misreported
as telemetry. Programmer errors (endpoint-policy violation of the fixed
constant) raise instead of becoming telemetry.

Parser-integration boundary

For every valid fixture payload, collect_zai_capacity(...) normalizes
identically (dataclass equality) to a direct
parse_zai_quota_response(decoded_payload, retrieved_at=...) call - asserted
for the known-windows, unknown-window and degraded-values fixtures. All
window/percentage/reset semantics remain in the pure parser.

Secret non-leak guarantees

A sweep over 27 scenarios (success, every transport failure class, every
malformed-credential class, and the duplicate-key classes) asserts every
synthetic secret is absent from snapshot.to_dict() JSON, repr(snapshot),
diagnostic fields, and captured stdout/stderr; production code emits no
stdout/stderr at all and no raw provider response is persisted. The safety
invariant is enforced by rejection before the transport layer: unsafe or
ambiguous credentials can never reach code that would build an exception
containing them; there is no reliance on catching-and-redacting.

Tests

tests/test_zai_acquisition.py: 69 tests, all network-free - mocked
transport seam (open_response), fake openers/responses, temporary auth
files, conspicuous synthetic secrets only (TEST_ONLY_*). The
reconnaissance helper tools/m1_zai_quota_recon.py remains untouched
evidence tooling and is not imported.

No test performs a live request, reads the real Kilo credential, or
requires the owner's secret.

Results

Total suite: 219 tests, all passing (150 baseline + 69 new)
basedpyright: 0 errors, 0 warnings, 0 notes (gate unchanged, no
suppressions, no baselines)
make check: exit 0

Explicit non-scope

No CLI status, provider aggregation, freshness caching, history, OpenAI/
Codex collector, Codex app-server, Ollama inspection, model selection,
capability ratings, scarcity calculations, REST, MCP, dashboard, model
execution, retry/backoff, endpoint configuration framework, credential
migration, OS keyring, new runtime dependencies (stdlib urllib/json
only), packaging metadata (U-008 untouched), and no live
credential-dependent integration test.

Closes #10 ## What this adds The production acquisition boundary for the Z.ai Coding Plan collector: the existing Kilo credential -> strict credential discovery -> fixed HTTPS quota GET -> bounded strict JSON decoding -> existing `parse_zai_quota_response(...)` -> `CapacitySnapshot`. The pure parser in `scarcity_router/providers/zai.py` is untouched; acquisition adds zero quota semantics of its own. ## Acquisition API ```python collect_zai_capacity( *, retrieved_at: str, # caller-supplied; no clock/freshness policy (U-003 untouched) auth_file: Path | None = None, # default: ~/.local/share/kilo/auth.json (evidenced Kilo location) ) -> CapacitySnapshot ``` The explicit `auth_file` exists for deterministic tests and controlled local configuration. The credential path and value never enter the snapshot. ## Credential-source contract - Supported layout: a JSON object keyed by provider ID; only the exact `zai-coding-plan` key is selected. The `key` value is inspected only after the target entry is established; unrelated provider entries are never read as candidates and never appear in output. - Evidenced shape required: `type == "api"` and a non-empty string `key`. No alternative fields (`token`, `value`, `apiKey`, `api_key`, `tokenValue`) and no token-like types are accepted. - The stored value is sent as-is: no `Bearer` prefix, no strip, no reformat. - Bounded read: 1 MiB (`MAX_AUTH_FILE_BYTES`); oversized, unreadable, missing, invalid-UTF-8 and invalid-JSON files all fail safely to `auth_required`. The filesystem path is never included in diagnostics. ### Credential header safety (security-review hardening) A credential is accepted only if it can be represented **byte-for-value** as an HTTP header value: a conservative printable-ASCII allowlist (space through tilde). CR, LF, CRLF, NUL, every other C0 control, DEL, C1 controls and values the stdlib latin-1 header encoding cannot represent are rejected during discovery - **before any request object or transport exists** - so no malformed credential can ever be formatted into a transport exception (`http.client` raises `ValueError("Illegal header value %r")` / `UnicodeEncodeError` containing the value). Accepted values (including leading/trailing spaces) are still transmitted exactly as stored; nothing is stripped, normalized or escaped. Unsafe credentials normalize to `auth_required` with zero transport calls. ### Duplicate-key ambiguity (security-review hardening) The auth document is parsed with a duplicate-rejecting JSON `object_pairs_hook`: **any duplicate object key anywhere in the document** fails closed to `auth_required` with zero requests - a repeated top-level `zai-coding-plan` entry, repeated `type`/`key` fields inside the target entry, and (documented conservative choice) duplicates in unrelated entries. Credential selection never trusts last-key-wins parser overwrite behavior, and neither candidate value can reach the transport or any output. ## Fixed endpoint and destination validation - `GET https://api.z.ai/api/monitor/usage/quota/limit` is a module constant; the public API exposes no endpoint override. - `is_approved_destination` (small, independently testable) rejects at minimum: non-HTTPS, wrong host, host-suffix tricks (`api.z.ai.example.com`), userinfo, explicit unexpected ports, query and fragments, wrong paths; malformed URLs fail closed. - Validation runs before the Authorization header is attached; if the fixed constant itself ever failed policy, a `RuntimeError` is raised before any header exists (programmer error, not disguised telemetry) - asserted by test that Authorization is never attached in that case. ## Redirect policy `HTTPRedirectHandler.redirect_request` is overridden to decline every 3xx (public `NoRedirect` handler; `open_response` builds the opener with it). Coverage of the real production boundary, network-free: - `NoRedirect.redirect_request` returns `None` for 301/302/303/307/308 and constructs no replacement request; - `open_response` obtains its opener via `urllib.request.build_opener` with exactly the `NoRedirect` handler and passes the original authenticated request object exactly once - the test fails if production ever switches to `urlopen(...)` or a default redirect-following opener. Mapping tests additionally assert per 3xx code: redirect not followed, zero second requests, credential never forwarded, snapshot contains no raw `Location`. ## Authorization behavior and bounds - Headers: `Authorization: <stored credential exactly>` and `Accept: application/json`; no other secret-bearing header. - Exactly one request, one credential use, no retries, no Bearer fallback. - Finite timeout: `TIMEOUT_SECONDS = 15`. - Response body bounded at 64 KiB (`MAX_BODY_BYTES`); oversized bodies are neither parsed nor persisted. Auth-file read bounded at 1 MiB. ## Failure/status mapping (v1 capacity contract) | Condition | status | diagnostic | | --- | --- | --- | | Credential missing/unreadable/malformed/wrong shape/unsafe header value/ambiguous | `auth_required` | `auth_required` | | HTTP 401 | `auth_required` | `auth_required` | | Success response with malformed UTF-8/JSON/empty body | `schema_changed` | `schema_changed` | | Connection/DNS/timeout/malformed HTTP protocol/mid-body failure | `unavailable` | `source_unavailable` | | Redirect rejected, oversized body, other unevidenced HTTP errors (403/429/5xx...) | `unknown` | `telemetry_unknown` | No raw transport exception escapes; invalid `retrieved_at` still fails through the typed capacity-contract validation instead of being misreported as telemetry. Programmer errors (endpoint-policy violation of the fixed constant) raise instead of becoming telemetry. ## Parser-integration boundary For every valid fixture payload, `collect_zai_capacity(...)` normalizes identically (dataclass equality) to a direct `parse_zai_quota_response(decoded_payload, retrieved_at=...)` call - asserted for the known-windows, unknown-window and degraded-values fixtures. All window/percentage/reset semantics remain in the pure parser. ## Secret non-leak guarantees A sweep over **27 scenarios** (success, every transport failure class, every malformed-credential class, and the duplicate-key classes) asserts every synthetic secret is absent from `snapshot.to_dict()` JSON, `repr(snapshot)`, diagnostic fields, and captured stdout/stderr; production code emits no stdout/stderr at all and no raw provider response is persisted. The safety invariant is enforced by rejection **before** the transport layer: unsafe or ambiguous credentials can never reach code that would build an exception containing them; there is no reliance on catching-and-redacting. ## Tests `tests/test_zai_acquisition.py`: 69 tests, all network-free - mocked transport seam (`open_response`), fake openers/responses, temporary auth files, conspicuous synthetic secrets only (`TEST_ONLY_*`). The reconnaissance helper `tools/m1_zai_quota_recon.py` remains untouched evidence tooling and is not imported. **No test performs a live request, reads the real Kilo credential, or requires the owner's secret.** ## Results - Total suite: 219 tests, all passing (150 baseline + 69 new) - basedpyright: 0 errors, 0 warnings, 0 notes (gate unchanged, no suppressions, no baselines) - `make check`: exit 0 ## Explicit non-scope No CLI `status`, provider aggregation, freshness caching, history, OpenAI/ Codex collector, Codex app-server, Ollama inspection, model selection, capability ratings, scarcity calculations, REST, MCP, dashboard, model execution, retry/backoff, endpoint configuration framework, credential migration, OS keyring, new runtime dependencies (stdlib `urllib`/`json` only), packaging metadata (U-008 untouched), and no live credential-dependent integration test.

Rows
Columns

M1: Implement secure Z.ai quota acquisition #11