Preserve or explicitly reject provider reasoning output semantics #158

New issue

Closed

opened 2026-09-28 16:43:07 +00:00 by kilo · 0 comments

kilo commented

2026-09-28 16:43:07 +00:00

Member

Defect (reproduced on canonical `develop` @ `6fec46ca82901a5fb8761909778c16d8322bbb97`, 2026-09-28)

The OpenAI-compatible translation core accepts reasoning controls on requests (D-055) and maps them deliberately per preset, but provider reasoning OUTPUT is not represented anywhere in the normalized execution seam. A backend can successfully return reasoning and Scarcity Router silently discards it while returning a successful response. Probe output on the exact base:

=== NON-STREAMING: deepseek-shaped reasoning_content ===
message fields: ['content', 'name', 'role', 'tool_call_id', 'tool_calls']
VERDICT: SILENTLY DISCARDED
=== NON-STREAMING: openrouter-shaped message.reasoning ===
VERDICT: SILENTLY DISCARDED
=== STREAMING: delta.reasoning_content / delta.reasoning ===
StreamFrameView fields: ['finish_reason', 'text_delta', 'tool_fragments', 'usage']
VERDICT: SILENTLY DISCARDED (both)
=== NON-STREAMING: reasoning-only response (content=null) ===
typed failure: TranslationError - provider message carries neither content nor tool_calls   (wrong failure: reasoning was present)
=== NON-STREAMING: unevidenced reasoning field on openai-api preset ===
parsed; reasoning silently tolerated + dropped

This violates the project's central translation rule: unsupported or unrepresentable semantics must never be silently discarded.

Affected source paths

scarcity_router/providers/openai_http_core.py — parse_chat_completion_response() reads only content/tool_calls; interpret_stream_frame() reads only delta.content/delta.tool_calls; StreamFrameView has no reasoning member; TranslationPolicy has no response-side reasoning fact.
scarcity_router/gateway_adapters.py — AdapterMessage has no reasoning member; CHUNK_KINDS has no reasoning-delta kind.
scarcity_router/gateway_openai.py — _message_payload/chunk_payload render no reasoning representation.
scarcity_router/providers/openai_http_adapter.py (server-direct) and scarcity_router/worker_local_translation.py (worker-loopback) — both consume the shared core, so both transports lose reasoning identically; the server-direct stream loop also assembles the final message without reasoning.
scarcity_router/worker_protocol.py — message_to_dict/message_from_dict cannot carry reasoning across the worker↔server boundary.
scarcity_router/providers/openai_http_presets.py — no preset carries a response-reasoning fact.

Provider evidence (official docs, fetched 2026-09-28)

DeepSeek (api-docs.deepseek.com/api/create-chat-completion): message.reasoning_content and delta.reasoning_content, "string nullable, for thinking mode only … the reasoning contents of the assistant message, before the final answer". Input reasoning_content exists only as a BETA Chat-Prefix-Completion feature on the beta base URL with prefix=true — not part of the evidenced production surface (response-only for us).
OpenRouter (openrouter.ai/docs/use-cases/reasoning-tokens): message.reasoning (string) with reasoning_content documented as an identical alias; streaming plaintext delta.reasoning; structured message/delta.reasoning_details arrays (summary/encrypted/text objects) — a structured shape this task deliberately does NOT represent.
Z.ai (docs.z.ai/api-reference/llm/chat-completion): message.reasoning_content (string), "Reasoning content, supported by GLM-4.5 series"; delta-level schema not separately documented on the retrieved reference (coding-endpoint parity remains PARTIAL, as the preset already records for reasoning controls).
OpenAI API / Ollama / generic: no documented reasoning-output field on the retrieved references (platform.openai.com chat completions exposes no reasoning text; docs.ollama.com/api/openai-compatibility documents reasoning effort controls only, "Reasoning summaries" only under the Responses API).

Semantic invariant (after this task)

Any provider reasoning output that an evidenced preset can produce must either be preserved through the normalized execution path and rendered to the client with documented semantics, or the response must fail explicitly with a typed translation/protocol-drift error. Silent loss is forbidden, in non-streaming AND streaming execution.

Design (frozen before implementation)

Normalized seam: AdapterMessage.reasoning: str | None (bounded like content; opaque provider output — never parsed, never routed on, never telemetry) and a dedicated reasoning_delta stream-chunk kind carrying its own bounded text; a reasoning delta is never a text_delta.
Translation policy: new closed policy fact reasoning_output_policy ∈ {none, reasoning_content, reasoning}. The preset decides which response shape is evidenced. Known reasoning-output field names (reasoning_content, reasoning, reasoning_details) in a response are: mapped when evidenced (strict string-or-null typing; OpenRouter's documented identical alias reconciled, conflicting values fail closed), typed drift when not evidenced — never flattened, never stringified, never silently discarded. No auto-detection from payload shape, no inference from hostname/model.
Public contract (additive): one documented client-facing representation — choices[0].message.reasoning_content (non-streaming) and choices[0].delta.reasoning_content (streaming), present only when reasoning was preserved (absence remains absence — no fabricated nulls/empties). reasoning_content is the ecosystem-standard name: defined natively by DeepSeek and Z.ai, and documented by OpenRouter as an identical alias of reasoning; LiteLLM's normalized representation is exactly reasoning_content with delta.reasoning mapped into it.
History/round-trip: response-only. Inbound message validation keeps its closed key set — a client echoing reasoning_content back receives the existing typed unknown_parameter 400 (never an unexplained drop). Matches DeepSeek's documented input restrictions; no evidenced preset supports reasoning re-injection.
Fail-closed details: wrong type → TranslationError (never stringified); two conflicting equivalent fields → fail closed; structured reasoning_details → typed drift; reasoning preserved independently alongside content and tool calls; reasoning-only terminal responses become representable if evidenced.
Streaming: deterministic reasoning-delta ordering; fragmented-SSE/UTF-8 boundary correctness inherited from the authoritative SSE parser (out of scope for replacement); no full-stream buffering of reasoning.
Worker transport: additive optional reasoning member on the assistant result message; worker protocol version bump per the documented v2 "one optional member" precedent, with the server accepting older peers; a negotiated-old session can never silently drop reasoning (new workers negotiate their own version or fail at hello, fail-closed).

Privacy/security

Reasoning content is potentially sensitive model output. It must not newly enter logs, audit notes, error messages, provider diagnostic notes, persisted configuration, SQLite audit metadata, routing state, metrics or telemetry (the audit record already structurally cannot carry content). Errors stay structural (parameter names, never values). Reasoning strings carry deterministic bounds consistent with existing assistant-content limits.

Reuse (focused; #150's broad audit not repeated)

LiteLLM v1.102.1 @ 1ceeefbf (MIT), litellm/types/utils.py: normalized reasoning_content on message and delta; delta.reasoning canonicalized into it. ADOPT the normalized semantics (not the code — pydantic framework idioms do not fit the stdlib translation core).
CLIProxyAPI @ 4a2c8186 (MIT), internal/thinking/summary.go and translators: per-provider reasoning-output field knowledge confined to translation edges. Corroborates keeping provider-specific knowledge at the policy edge; no code reuse.
Both were REJECTed as machinery in #150 (silent-drop design center); that verdict stands — only the normalized-field semantics are adopted, and our fail-closed policy discipline is strictly stronger than both.

Scope

Translation core + policy + presets + normalized seam + gateway rendering + worker protocol member + tests + docs/decision record (next decision number from current develop; D-060/D-061 are reserved by open #156/#157). Both transports (server-direct, worker-loopback) preserve reasoning through the ONE shared core; no second implementation.

Non-goals (explicitly out of scope)

Z.ai CREDIT_LIMIT; worker chunk-buffer hardening; HTTP transport changes; SSE parser replacement; invalid-UTF-8 typing cleanup; Responses API; Anthropic Messages; new providers; Codex schema automation; client-owned tool execution; generic content-block abstractions; the Codex (M06) adapter's item/reasoning/* summary handling (separate protocol + evidence path, recorded here as adjacent observed behavior, unchanged); structured reasoning_details representation; the #150 deferred items (HTTP early-response/write-error refinement, connect-phase error classification, worker chunk bounds).

Acceptance criteria

Failing regression reproduced on base flips green; required test matrix covered: translation-core non-streaming (8 cases), translation-core streaming (9 cases), normalized seam, gateway rendering (non-streaming + streaming + synthesis), transport parity (server-direct + worker-loopback), tool-call interaction (6 cases), unevidenced-shape drift, privacy (no reasoning in logs/audit/diagnostics), representative-client compatibility.
Full gate green (uv sync --only-dev, make check, uv run basedpyright, git diff --check).
Decision record in docs/decisions.md; execution-surface/providers/worker-protocol docs updated in place.
Independent exact-head adversarial review (12-point checklist incl. UX impact); PR open targeting develop, never merged by agents.

## Defect (reproduced on canonical `develop` @ `6fec46ca82901a5fb8761909778c16d8322bbb97`, 2026-09-28) The OpenAI-compatible translation core accepts reasoning controls on requests (D-055) and maps them deliberately per preset, but provider reasoning OUTPUT is not represented anywhere in the normalized execution seam. A backend can successfully return reasoning and Scarcity Router silently discards it while returning a successful response. Probe output on the exact base: ``` === NON-STREAMING: deepseek-shaped reasoning_content === message fields: ['content', 'name', 'role', 'tool_call_id', 'tool_calls'] VERDICT: SILENTLY DISCARDED === NON-STREAMING: openrouter-shaped message.reasoning === VERDICT: SILENTLY DISCARDED === STREAMING: delta.reasoning_content / delta.reasoning === StreamFrameView fields: ['finish_reason', 'text_delta', 'tool_fragments', 'usage'] VERDICT: SILENTLY DISCARDED (both) === NON-STREAMING: reasoning-only response (content=null) === typed failure: TranslationError - provider message carries neither content nor tool_calls (wrong failure: reasoning was present) === NON-STREAMING: unevidenced reasoning field on openai-api preset === parsed; reasoning silently tolerated + dropped ``` This violates the project's central translation rule: unsupported or unrepresentable semantics must never be silently discarded. ## Affected source paths - `scarcity_router/providers/openai_http_core.py` — `parse_chat_completion_response()` reads only `content`/`tool_calls`; `interpret_stream_frame()` reads only `delta.content`/`delta.tool_calls`; `StreamFrameView` has no reasoning member; `TranslationPolicy` has no response-side reasoning fact. - `scarcity_router/gateway_adapters.py` — `AdapterMessage` has no reasoning member; `CHUNK_KINDS` has no reasoning-delta kind. - `scarcity_router/gateway_openai.py` — `_message_payload`/`chunk_payload` render no reasoning representation. - `scarcity_router/providers/openai_http_adapter.py` (server-direct) and `scarcity_router/worker_local_translation.py` (worker-loopback) — both consume the shared core, so both transports lose reasoning identically; the server-direct stream loop also assembles the final message without reasoning. - `scarcity_router/worker_protocol.py` — `message_to_dict`/`message_from_dict` cannot carry reasoning across the worker↔server boundary. - `scarcity_router/providers/openai_http_presets.py` — no preset carries a response-reasoning fact. ## Provider evidence (official docs, fetched 2026-09-28) - **DeepSeek** (`api-docs.deepseek.com/api/create-chat-completion`): `message.reasoning_content` and `delta.reasoning_content`, "string nullable, for thinking mode only … the reasoning contents of the assistant message, before the final answer". Input `reasoning_content` exists only as a BETA Chat-Prefix-Completion feature on the beta base URL with `prefix=true` — not part of the evidenced production surface (response-only for us). - **OpenRouter** (`openrouter.ai/docs/use-cases/reasoning-tokens`): `message.reasoning` (string) with `reasoning_content` documented as an identical alias; streaming plaintext `delta.reasoning`; structured `message/delta.reasoning_details` arrays (summary/encrypted/text objects) — a structured shape this task deliberately does NOT represent. - **Z.ai** (`docs.z.ai/api-reference/llm/chat-completion`): `message.reasoning_content` (string), "Reasoning content, supported by GLM-4.5 series"; delta-level schema not separately documented on the retrieved reference (coding-endpoint parity remains PARTIAL, as the preset already records for reasoning controls). - **OpenAI API / Ollama / generic**: no documented reasoning-output field on the retrieved references (`platform.openai.com` chat completions exposes no reasoning text; `docs.ollama.com/api/openai-compatibility` documents reasoning effort controls only, "Reasoning summaries" only under the Responses API). ## Semantic invariant (after this task) Any provider reasoning output that an evidenced preset can produce must either be preserved through the normalized execution path and rendered to the client with documented semantics, or the response must fail explicitly with a typed translation/protocol-drift error. Silent loss is forbidden, in non-streaming AND streaming execution. ## Design (frozen before implementation) 1. **Normalized seam**: `AdapterMessage.reasoning: str | None` (bounded like content; opaque provider output — never parsed, never routed on, never telemetry) and a dedicated `reasoning_delta` stream-chunk kind carrying its own bounded text; a reasoning delta is never a `text_delta`. 2. **Translation policy**: new closed policy fact `reasoning_output_policy` ∈ {`none`, `reasoning_content`, `reasoning`}. The preset decides which response shape is evidenced. Known reasoning-output field names (`reasoning_content`, `reasoning`, `reasoning_details`) in a response are: mapped when evidenced (strict string-or-null typing; OpenRouter's documented identical alias reconciled, conflicting values fail closed), typed drift when not evidenced — never flattened, never stringified, never silently discarded. No auto-detection from payload shape, no inference from hostname/model. 3. **Public contract (additive)**: one documented client-facing representation — `choices[0].message.reasoning_content` (non-streaming) and `choices[0].delta.reasoning_content` (streaming), present only when reasoning was preserved (absence remains absence — no fabricated nulls/empties). `reasoning_content` is the ecosystem-standard name: defined natively by DeepSeek and Z.ai, and documented by OpenRouter as an identical alias of `reasoning`; LiteLLM's normalized representation is exactly `reasoning_content` with `delta.reasoning` mapped into it. 4. **History/round-trip: response-only.** Inbound message validation keeps its closed key set — a client echoing `reasoning_content` back receives the existing typed `unknown_parameter` 400 (never an unexplained drop). Matches DeepSeek's documented input restrictions; no evidenced preset supports reasoning re-injection. 5. **Fail-closed details**: wrong type → `TranslationError` (never stringified); two conflicting equivalent fields → fail closed; structured `reasoning_details` → typed drift; reasoning preserved independently alongside content and tool calls; reasoning-only terminal responses become representable if evidenced. 6. **Streaming**: deterministic reasoning-delta ordering; fragmented-SSE/UTF-8 boundary correctness inherited from the authoritative SSE parser (out of scope for replacement); no full-stream buffering of reasoning. 7. **Worker transport**: additive optional `reasoning` member on the assistant result message; worker protocol version bump per the documented v2 "one optional member" precedent, with the server accepting older peers; a negotiated-old session can never silently drop reasoning (new workers negotiate their own version or fail at hello, fail-closed). ## Privacy/security Reasoning content is potentially sensitive model output. It must not newly enter logs, audit notes, error messages, provider diagnostic notes, persisted configuration, SQLite audit metadata, routing state, metrics or telemetry (the audit record already structurally cannot carry content). Errors stay structural (parameter names, never values). Reasoning strings carry deterministic bounds consistent with existing assistant-content limits. ## Reuse (focused; #150's broad audit not repeated) - LiteLLM v1.102.1 @ `1ceeefbf` (MIT), `litellm/types/utils.py`: normalized `reasoning_content` on message and delta; `delta.reasoning` canonicalized into it. **ADOPT the normalized semantics** (not the code — pydantic framework idioms do not fit the stdlib translation core). - CLIProxyAPI @ `4a2c8186` (MIT), `internal/thinking/summary.go` and translators: per-provider reasoning-output field knowledge confined to translation edges. Corroborates keeping provider-specific knowledge at the policy edge; no code reuse. - Both were REJECTed as machinery in #150 (silent-drop design center); that verdict stands — only the normalized-field semantics are adopted, and our fail-closed policy discipline is strictly stronger than both. ## Scope Translation core + policy + presets + normalized seam + gateway rendering + worker protocol member + tests + docs/decision record (next decision number from current `develop`; D-060/D-061 are reserved by open #156/#157). Both transports (server-direct, worker-loopback) preserve reasoning through the ONE shared core; no second implementation. ## Non-goals (explicitly out of scope) Z.ai `CREDIT_LIMIT`; worker chunk-buffer hardening; HTTP transport changes; SSE parser replacement; invalid-UTF-8 typing cleanup; Responses API; Anthropic Messages; new providers; Codex schema automation; client-owned tool execution; generic content-block abstractions; the Codex (M06) adapter's `item/reasoning/*` summary handling (separate protocol + evidence path, recorded here as adjacent observed behavior, unchanged); structured `reasoning_details` representation; the #150 deferred items (HTTP early-response/write-error refinement, connect-phase error classification, worker chunk bounds). ## Acceptance criteria - Failing regression reproduced on base flips green; required test matrix covered: translation-core non-streaming (8 cases), translation-core streaming (9 cases), normalized seam, gateway rendering (non-streaming + streaming + synthesis), transport parity (server-direct + worker-loopback), tool-call interaction (6 cases), unevidenced-shape drift, privacy (no reasoning in logs/audit/diagnostics), representative-client compatibility. - Full gate green (`uv sync --only-dev`, `make check`, `uv run basedpyright`, `git diff --check`). - Decision record in `docs/decisions.md`; execution-surface/providers/worker-protocol docs updated in place. - Independent exact-head adversarial review (12-point checklist incl. UX impact); PR open targeting `develop`, never merged by agents.