Creatidy/scarcity-router

Fork 0

M03: OpenAI-compatible gateway and execution coordinator #88

New issue

Closed

opened 2026-09-19 13:06:31 +00:00 by kilo · 1 comment

kilo commented

2026-09-19 13:06:31 +00:00

Member

Goal

Expose a real execution endpoint to compatible clients without taking over client-side orchestration.

Module of the execution-gateway program; architecture and contracts come from A0 (#85).

User value

Any OpenAI SDK client (CLI, IDE, agent) can point at Scarcity Router and run multi-turn tool-using conversations against the best authorized backend, with honest capability handling, cancellation and usage accounting.

Relationship to current code

New module(s) beside the existing application layer; reuses the routing core (M02), registry/state (M01) and collector eligibility (D-039) rather than duplicating them.
The existing loopback REST v1 adapter (server.py) stays frozen; the gateway is a NEW separately versioned surface (A0 decision) served by the same server process as the control API and UI (M09).

Scope

GET /v1/models and POST /v1/chat/completions (minimum ingress), streaming SSE, and request capability validation before inference.
Integration with the authoritative routing core; support pinned-target execution and required session isolation.
Minimal execution lifecycle: admission, bounded concurrency, timeout, cancellation (client disconnect propagates to the selected backend where supported), usage/accounting, safe diagnostics.
Treat the Responses API as a later explicit sub-scope of this module with a documented supported subset; never expose a fake /v1/responses that silently drops unsupported semantics.
Per-adapter/adapter-version/model OpenAI compatibility matrix: roles and conversation history, streaming, tool_calls, tool results, structured output, reasoning controls, context limits, error semantics, usage reporting, cancellation. An agentic CLI backend is NOT automatically an OpenAI-compatible backend; concatenating messages into a text prompt is NOT sufficient compatibility.
Unsupported request capabilities are rejected before inference or routed only to an actually compatible and authorized backend.
Tools supplied by the client return to the CLIENT as tool_calls; the router never automatically executes client-provided tools as a local shell command, Codex MCP, ZCode MCP or any local tool.
An execution mode with undefined or unbounded start time must not silently replace a synchronous HTTP request; no general task scheduler in this program.
After a response stream has started, the backend is never silently replaced. Retries after ambiguous execution state must not blindly cause duplicate inference consumption; exactly-once execution is not promised. One external request may cause multiple internal provider calls — usage/accounting represents this honestly.

Non-goals

No client-side orchestration takeover; no autonomous fallback execution; no repository/source-code access; no issue-to-PR orchestration; no generic LLM-gateway features beyond the documented surface.

Contracts / interfaces affected

New versioned OpenAI-compatible execution contract (A0); machine-interface v1 unchanged (existing /v1/* untouched); audit metadata per A0's contract.

Dependencies

Depends on: A0 (#85), M01 (#86), M02 (#87).
Feeds: M04 (generic adapter plugs in here), M05 (worker dispatch), M09 (UI/diagnostics), M10 (E2E).

Acceptance criteria

A standard OpenAI SDK and at least one representative IDE/client complete a multi-turn tool round-trip on a backend that actually supports it.
Missing capabilities are not silently ignored.
Client disconnect/cancellation propagates to the selected execution backend where supported.
No silent backend replacement after a response stream has started; no blind duplicate inference consumption on ambiguous retries.
Usage/accounting honestly represents multi-call fan-out; audit metadata per A0 without prompt/response contents.
Security: authenticated inference-client identity; request-body, context/output, concurrency, execution-time and spending limits; no bearer secrets in URLs; router-loop protection (a router's own endpoint must not silently serve as a backend for another router instance); no prompt/response logging by default; redacted diagnostics.
Full validation gate green; deterministic fixture tests; paid-provider tests are explicit opt-in only.

Security considerations

This is the ingress boundary and owns the first executable vertical slice's security posture: authentication, limits, session/filesystem/tool isolation enforced before any backend dispatch, all per A0's threat model.

Risks

Compatibility-matrix evidence burden across backends — mitigated by per-adapter matrices with explicit UNKNOWN instead of optimistic defaults.
Streaming/cancellation semantics differing per backend — mitigated by the execution contract's explicit per-backend capability reporting.

Evidence still required

Representative client versions for acceptance (OpenAI SDK, one IDE/client), dated.
Backend capability evidence (Ollama OpenAI-compat doc, Z.ai coding endpoint, Codex app-server protocol) — dated and versioned; re-verify at implementation time.

## Goal Expose a real execution endpoint to compatible clients without taking over client-side orchestration. Module of the execution-gateway program; architecture and contracts come from A0 (#85). ## User value Any OpenAI SDK client (CLI, IDE, agent) can point at Scarcity Router and run multi-turn tool-using conversations against the best authorized backend, with honest capability handling, cancellation and usage accounting. ## Relationship to current code - New module(s) beside the existing application layer; reuses the routing core (M02), registry/state (M01) and collector eligibility (D-039) rather than duplicating them. - The existing loopback REST v1 adapter (`server.py`) stays frozen; the gateway is a NEW separately versioned surface (A0 decision) served by the same server process as the control API and UI (M09). ## Scope - `GET /v1/models` and `POST /v1/chat/completions` (minimum ingress), streaming SSE, and request capability validation before inference. - Integration with the authoritative routing core; support pinned-target execution and required session isolation. - Minimal execution lifecycle: admission, bounded concurrency, timeout, cancellation (client disconnect propagates to the selected backend where supported), usage/accounting, safe diagnostics. - Treat the Responses API as a later explicit sub-scope of this module with a documented supported subset; never expose a fake `/v1/responses` that silently drops unsupported semantics. - Per-adapter/adapter-version/model OpenAI compatibility matrix: roles and conversation history, streaming, `tool_calls`, tool results, structured output, reasoning controls, context limits, error semantics, usage reporting, cancellation. An agentic CLI backend is NOT automatically an OpenAI-compatible backend; concatenating messages into a text prompt is NOT sufficient compatibility. - Unsupported request capabilities are rejected before inference or routed only to an actually compatible and authorized backend. - Tools supplied by the client return to the CLIENT as `tool_calls`; the router never automatically executes client-provided tools as a local shell command, Codex MCP, ZCode MCP or any local tool. - An execution mode with undefined or unbounded start time must not silently replace a synchronous HTTP request; no general task scheduler in this program. - After a response stream has started, the backend is never silently replaced. Retries after ambiguous execution state must not blindly cause duplicate inference consumption; exactly-once execution is not promised. One external request may cause multiple internal provider calls — usage/accounting represents this honestly. ## Non-goals - No client-side orchestration takeover; no autonomous fallback execution; no repository/source-code access; no issue-to-PR orchestration; no generic LLM-gateway features beyond the documented surface. ## Contracts / interfaces affected New versioned OpenAI-compatible execution contract (A0); machine-interface v1 unchanged (existing `/v1/*` untouched); audit metadata per A0's contract. ## Dependencies - Depends on: A0 (#85), M01 (#86), M02 (#87). - Feeds: M04 (generic adapter plugs in here), M05 (worker dispatch), M09 (UI/diagnostics), M10 (E2E). ## Acceptance criteria - [ ] A standard OpenAI SDK and at least one representative IDE/client complete a multi-turn tool round-trip on a backend that actually supports it. - [ ] Missing capabilities are not silently ignored. - [ ] Client disconnect/cancellation propagates to the selected execution backend where supported. - [ ] No silent backend replacement after a response stream has started; no blind duplicate inference consumption on ambiguous retries. - [ ] Usage/accounting honestly represents multi-call fan-out; audit metadata per A0 without prompt/response contents. - [ ] Security: authenticated inference-client identity; request-body, context/output, concurrency, execution-time and spending limits; no bearer secrets in URLs; router-loop protection (a router's own endpoint must not silently serve as a backend for another router instance); no prompt/response logging by default; redacted diagnostics. - [ ] Full validation gate green; deterministic fixture tests; paid-provider tests are explicit opt-in only. ## Security considerations This is the ingress boundary and owns the first executable vertical slice's security posture: authentication, limits, session/filesystem/tool isolation enforced before any backend dispatch, all per A0's threat model. ## Risks - Compatibility-matrix evidence burden across backends — mitigated by per-adapter matrices with explicit UNKNOWN instead of optimistic defaults. - Streaming/cancellation semantics differing per backend — mitigated by the execution contract's explicit per-backend capability reporting. ## Evidence still required - Representative client versions for acceptance (OpenAI SDK, one IDE/client), dated. - Backend capability evidence (Ollama OpenAI-compat doc, Z.ai coding endpoint, Codex app-server protocol) — dated and versioned; re-verify at implementation time.

kilo referenced this issue

2026-09-19 13:11:01 +00:00

A0: Architecture, decisions, and contracts for the optional execution gateway #85

kilo added the

enhancement

label

2026-09-19 13:11:40 +00:00

kilo added a new dependency

2026-09-19 13:11:59 +00:00

#85 A0: Architecture, decisions, and contracts for the optional execution gateway

kilo added a new dependency

2026-09-19 13:12:00 +00:00

#86 M01: Resource registry, state, and collectors for executable resources

kilo added a new dependency

2026-09-19 13:12:00 +00:00

#87 M02: Routing core, policy, and client-requirement binding for executable targets

kilo added a new dependency

2026-09-19 13:12:00 +00:00

#89 M04: Generic OpenAI-compatible HTTP adapter and Ollama integration

kilo added a new dependency

2026-09-19 13:12:00 +00:00

#90 M05: Native worker, pairing, and execution transport

kilo added a new dependency

2026-09-19 13:12:17 +00:00

#94 M09: Configuration, web UX, and diagnostics

kilo added a new dependency

2026-09-19 13:12:17 +00:00

#95 M10: Distribution, installation, update, and end-to-end acceptance

kilo referenced this issue

2026-09-19 13:45:55 +00:00

docs(architecture): execution-gateway program map, decisions and contracts (A0, issue #85) #96

kilo referenced this issue

2026-09-19 16:36:45 +00:00

docs(architecture): execution-gateway program map, decisions and contracts (A0, issue #85) #96

kilo referenced this issue

2026-09-19 19:52:09 +00:00

M01: Resource registry, state, and collectors for executable resources (issue #86) #99