Selector: Make reasoning effort first-class and onboard Terra #58

Merged

adrian.tkacz merged 2 commits from selector/reasoning-effort-terra into develop

2026-09-09 19:32:46 +00:00

kilo commented

2026-09-08 18:21:13 +00:00

Member

Closes #57 when human-reviewed and merged.

Problem and correction

The original four-configuration set reproduces routine_coding -> Luna Max and deep_coding -> Sol High (ScenarioTests.test_original_configuration_set_reproduces_effort_gap). Opaque variants could not express a separately calibrated lower-effort choice, and Terra was absent.

Catalog v2 (updated_on 2026-09-08) adds explicit ModelCatalogEntry.reasoning_effort. Vocabulary/order: none < low < medium < high < xhigh < max. Construction/deserialization reject unknown values; reasoning support true requires explicit effort, while false/unknown support requires null/absent. Null is not the real string setting "none". External reasoning-capable catalogs require explicit migration/calibration, never automatic inference from old variant strings.

ModelIdentity stays provider/model/variant. Variant remains opaque: the selector reads the catalog effort field, not display/model/provider names or variant spelling. Conflicting lexical-variant tests prove this. CandidateEvaluation and SelectionDecision serialized field sets are unchanged; current variants remain public configuration identifiers. D-032 records the compatibility boundary and supersedes only the prior ranking/catalog scope and implementation-status claims.

Calibrated configurations

Vectors are reasoning / coding / scientific_methodological / writing_editorial / tool_use / translation_multilingual:

Configuration	Vector	Change
Luna Medium	3,4,3,5,5,4	New
Luna Max	4,4,3,5,5,4	Preserved; explicit max
Terra Medium	4,5,4,4,5,4	New
Sol Medium	5,5,4,5,5,4	New
Sol High	5,5,5,5,5,5	Preserved; explicit high
GLM-5.3 Max	5,5,4,4,5,4	Preserved; explicit max
GLM-5.3-Flash Max	4,4,3,4,5,4	Preserved; explicit max

Exactly three additions, no automatic vector cloning and no other effort configurations. All new assessments carry medium confidence, per-dimension rationale, date and explicit owner-approved calibration provenance.

Terra uses architect-supplied accepted GPT-5.6 launch/developer evidence with the existing source identifiers https://openai.com/index/gpt-5-6/ and https://developers.openai.com/api/docs/models, plus accepted_reasoning_effort_calibration_2026-09-08_issue_57_D-032. This is curated evidence, not a new benchmark or web audit. Properties: 1,050,000 input, 128,000 output, tools/vision/reasoning true, openai/codex binding. Family version date remains the evidenced 2026-07-09.

Exact ranking

After unchanged hard/capability/capacity/reservation gates:

known capacity before degraded unknown -> scarcity penalty -> capability margin -> LOWEST REASONING EFFORT -> explicit preference -> stable identity.

Unconfigured effort uses an explicit typed comparison state after known effort, with no numeric intensity sentinel. Lower effort cannot rescue capability failure, override scarcity or weaken capability margin. Tests in RankingTests prove scarcity-before-margin, scarcity-before-effort, margin-before-effort, all adjacent effort boundaries plus High-before-Max, effort-before-preference with lexically conflicting opaque variants, preference-before-identity, and capability rejection.

Deterministic outcomes

OpenAI-side results when its capacity is available and Z.ai does not win on scarcity:

Profile	Before	After
routine_coding	Luna Max	Luna Medium
deep_coding	Sol High	Terra Medium
scientific_review	Sol High	Sol High
orchestration	Luna Max	Luna Max
translation	Sol High	Sol High

Deep-coding alternatives are Sol Medium then Sol High. Luna variants fail coding 4 < 5; Luna Medium also fails reasoning 3 < 4. Terra/Sol Medium fail scientific/translation specialist minima at 4 < 5. Orchestration excludes Luna Medium on reasoning and Terra Medium on writing. All eight existing profile eligible sets are pinned without changing minima.

Capacity and interfaces

All five OpenAI configurations have identical scarcity assessments for the shared openai/codex observation, verified by test_all_openai_efforts_share_identical_capacity_observation. No effort-specific quota, API-price routing or penalty is invented. GLM ratings/bindings are preserved; Z.ai still wins when less scarce, and exhausted coding-plan capacity still excludes GLM candidates with capacity_exhausted evidence.

Direct == CLI == REST == MCP passes, including the five-profile test SelectTests.test_five_effort_profiles_match_direct_cli_rest_and_mcp_v1 and existing status/select/simulation parity. REST paths, schema_version 1, MCP tool names/input schemas/envelopes, CLI JSON shapes and error vocabulary are unchanged. No duplicate interface implementation. docs/m3-acceptance.md is unchanged; historical M3 PASS remains valid.

The only model-policy.json change is descriptive reasoning_effort_policy.selector_support: not_implemented -> calibrated_configurations, avoiding a direct documentation contradiction. Policy v5 fields, version, profile minima and dated role assignments are unchanged.

Live sanity

Successful authorized recovery on 2026-09-08: one real collection reused unchanged in memory through the existing CLI injection seam for status and all five selections. OpenAI status ok, known/plentiful applicable capacity; Z.ai status ok, applicable capacity unavailable. Five selections match the After column, all non-degraded. No exact personal quota values, credentials or raw provider data retained; model execution NO.

The initial attempt failed in the harness after collection because cli.main does not accept stderr. The human explicitly authorized exactly one corrected pass (recorded in issue #57); the corrected harness was first verified with synthetic data. No further live retries or recollection occurred during the successful pass.

Validation and Git safety

uv sync --only-dev: PASS.
make check: 901 tests PASS; basedpyright 0 errors, 0 warnings, 0 notes.
Separate uv run basedpyright: 0 errors, 0 warnings, 0 notes.
git diff --check and staged whitespace check: PASS.
Clean worktree; one task commit; zero merge commits in origin/develop..HEAD.
Required core/catalog/policy branch diff inspected; provider collectors, capacity, scarcity, policy primitives and interface implementations unchanged.
Base develop: ea7332cfb5, containing M3c and merged M3 closeout #55.
Head/remote feature: ec52d6284f.
origin/develop before push: ea7332cfb5.
origin/develop after push: ea7332cfb5.
Pushed exactly HEAD:refs/heads/selector/reasoning-effort-terra to Forgejo; no GitHub write.

Human gate and non-scope

One agent/session, no subagents, no model switching, no internal reviewer loop. Assignment override: GPT-6 Astra / low requested; runtime effort independently unobservable, not falsely attested. Human freezes this head for one independent external review. Leave OPEN and UNMERGED.

Astra onboarding NOT DONE; issue #49 UNCHANGED; M4 NOT STARTED; provider collectors UNCHANGED; capacity v3 UNCHANGED; scarcity formula UNCHANGED; model execution NO.

Closes #57 when human-reviewed and merged. ## Problem and correction The original four-configuration set reproduces routine_coding -> Luna Max and deep_coding -> Sol High (ScenarioTests.test_original_configuration_set_reproduces_effort_gap). Opaque variants could not express a separately calibrated lower-effort choice, and Terra was absent. Catalog v2 (updated_on 2026-09-08) adds explicit ModelCatalogEntry.reasoning_effort. Vocabulary/order: none < low < medium < high < xhigh < max. Construction/deserialization reject unknown values; reasoning support true requires explicit effort, while false/unknown support requires null/absent. Null is not the real string setting "none". External reasoning-capable catalogs require explicit migration/calibration, never automatic inference from old variant strings. ModelIdentity stays provider/model/variant. Variant remains opaque: the selector reads the catalog effort field, not display/model/provider names or variant spelling. Conflicting lexical-variant tests prove this. CandidateEvaluation and SelectionDecision serialized field sets are unchanged; current variants remain public configuration identifiers. D-032 records the compatibility boundary and supersedes only the prior ranking/catalog scope and implementation-status claims. ## Calibrated configurations Vectors are reasoning / coding / scientific_methodological / writing_editorial / tool_use / translation_multilingual: | Configuration | Vector | Change | | --- | --- | --- | | Luna Medium | 3,4,3,5,5,4 | New | | Luna Max | 4,4,3,5,5,4 | Preserved; explicit max | | Terra Medium | 4,5,4,4,5,4 | New | | Sol Medium | 5,5,4,5,5,4 | New | | Sol High | 5,5,5,5,5,5 | Preserved; explicit high | | GLM-5.3 Max | 5,5,4,4,5,4 | Preserved; explicit max | | GLM-5.3-Flash Max | 4,4,3,4,5,4 | Preserved; explicit max | Exactly three additions, no automatic vector cloning and no other effort configurations. All new assessments carry medium confidence, per-dimension rationale, date and explicit owner-approved calibration provenance. Terra uses architect-supplied accepted GPT-5.6 launch/developer evidence with the existing source identifiers https://openai.com/index/gpt-5-6/ and https://developers.openai.com/api/docs/models, plus accepted_reasoning_effort_calibration_2026-09-08_issue_57_D-032. This is curated evidence, not a new benchmark or web audit. Properties: 1,050,000 input, 128,000 output, tools/vision/reasoning true, openai/codex binding. Family version date remains the evidenced 2026-07-09. ## Exact ranking After unchanged hard/capability/capacity/reservation gates: known capacity before degraded unknown -> scarcity penalty -> capability margin -> LOWEST REASONING EFFORT -> explicit preference -> stable identity. Unconfigured effort uses an explicit typed comparison state after known effort, with no numeric intensity sentinel. Lower effort cannot rescue capability failure, override scarcity or weaken capability margin. Tests in RankingTests prove scarcity-before-margin, scarcity-before-effort, margin-before-effort, all adjacent effort boundaries plus High-before-Max, effort-before-preference with lexically conflicting opaque variants, preference-before-identity, and capability rejection. ## Deterministic outcomes OpenAI-side results when its capacity is available and Z.ai does not win on scarcity: | Profile | Before | After | | --- | --- | --- | | routine_coding | Luna Max | Luna Medium | | deep_coding | Sol High | Terra Medium | | scientific_review | Sol High | Sol High | | orchestration | Luna Max | Luna Max | | translation | Sol High | Sol High | Deep-coding alternatives are Sol Medium then Sol High. Luna variants fail coding 4 < 5; Luna Medium also fails reasoning 3 < 4. Terra/Sol Medium fail scientific/translation specialist minima at 4 < 5. Orchestration excludes Luna Medium on reasoning and Terra Medium on writing. All eight existing profile eligible sets are pinned without changing minima. ## Capacity and interfaces All five OpenAI configurations have identical scarcity assessments for the shared openai/codex observation, verified by test_all_openai_efforts_share_identical_capacity_observation. No effort-specific quota, API-price routing or penalty is invented. GLM ratings/bindings are preserved; Z.ai still wins when less scarce, and exhausted coding-plan capacity still excludes GLM candidates with capacity_exhausted evidence. Direct == CLI == REST == MCP passes, including the five-profile test SelectTests.test_five_effort_profiles_match_direct_cli_rest_and_mcp_v1 and existing status/select/simulation parity. REST paths, schema_version 1, MCP tool names/input schemas/envelopes, CLI JSON shapes and error vocabulary are unchanged. No duplicate interface implementation. docs/m3-acceptance.md is unchanged; historical M3 PASS remains valid. The only model-policy.json change is descriptive reasoning_effort_policy.selector_support: not_implemented -> calibrated_configurations, avoiding a direct documentation contradiction. Policy v5 fields, version, profile minima and dated role assignments are unchanged. ## Live sanity Successful authorized recovery on 2026-09-08: one real collection reused unchanged in memory through the existing CLI injection seam for status and all five selections. OpenAI status ok, known/plentiful applicable capacity; Z.ai status ok, applicable capacity unavailable. Five selections match the After column, all non-degraded. No exact personal quota values, credentials or raw provider data retained; model execution NO. The initial attempt failed in the harness after collection because cli.main does not accept stderr. The human explicitly authorized exactly one corrected pass (recorded in issue #57); the corrected harness was first verified with synthetic data. No further live retries or recollection occurred during the successful pass. ## Validation and Git safety - uv sync --only-dev: PASS. - make check: 901 tests PASS; basedpyright 0 errors, 0 warnings, 0 notes. - Separate uv run basedpyright: 0 errors, 0 warnings, 0 notes. - git diff --check and staged whitespace check: PASS. - Clean worktree; one task commit; zero merge commits in origin/develop..HEAD. - Required core/catalog/policy branch diff inspected; provider collectors, capacity, scarcity, policy primitives and interface implementations unchanged. - Base develop: ea7332cfb5f926c5da19fa615b9a8792f518c5a9, containing M3c and merged M3 closeout #55. - Head/remote feature: ec52d6284fbc064e0edccbe7d0eef3b72d0d50cc. - origin/develop before push: ea7332cfb5f926c5da19fa615b9a8792f518c5a9. - origin/develop after push: ea7332cfb5f926c5da19fa615b9a8792f518c5a9. - Pushed exactly HEAD:refs/heads/selector/reasoning-effort-terra to Forgejo; no GitHub write. ## Human gate and non-scope One agent/session, no subagents, no model switching, no internal reviewer loop. Assignment override: GPT-6 Astra / low requested; runtime effort independently unobservable, not falsely attested. Human freezes this head for one independent external review. Leave OPEN and UNMERGED. Astra onboarding NOT DONE; issue #49 UNCHANGED; M4 NOT STARTED; provider collectors UNCHANGED; capacity v3 UNCHANGED; scarcity formula UNCHANGED; model execution NO.

kilo added 1 commit

2026-09-08 18:21:13 +00:00