M2c: Add calibrated model catalog and task profiles #37

Merged

creatidy merged 2 commits from m2c-model-calibration into main

2026-09-06 11:49:18 +00:00

creatidy commented

2026-09-06 11:14:59 +00:00

(Migrated from github.com)

Closes #36

Summary

M2c populates the first accepted content for the frozen M2a/M2b contracts (D-025, resolving U-006). It does not implement scarcity, reservations, capacity aggregation or selection.

Catalog artifact (`model-catalog.json`, version 1)

Exactly four routing identities, generated and round-tripped through the M2b types (ModelCatalog.from_dict), serialized deterministically in (provider, model, variant) order:

Identity	Display name	Version date
`openai/gpt-5.6-luna/max`	GPT-5.6 Luna Max	2026-07-09
`openai/gpt-5.6-sol/high`	GPT-5.6 Sol High	2026-07-09
`zai/glm-5.3/max`	GLM-5.3 Max	2026-08-14
`zai/glm-5.3-flash/max`	GLM-5.3-Flash Max	2026-08-26

Curated six-dimensional ratings on the frozen 1..5 scale (Luna 4/4/3/5/5/4, Sol 5/5/5/5/5/5, GLM-5.3 5/5/4/4/5/4, Flash 4/4/3/4/5/4) — the accepted calibration, not autonomously modified.
Every known rating carries ≥1 EvidenceRef, coarse confidence, assessed_on = 2026-09-06 and a rationale. No HumanOverride is used for initial curation.
Tri-state hard properties (evidenced, first-party): GLM-5.3 output_tokens = 128000 and supports_vision = false (explicit documented text-only inputs — a known negative serialized explicitly, not an omitted unknown); GLM-5.3-Flash output_tokens = 128000 with vision true; Flash reasoning mode (reasoning_effort: max, always-on thinking) is first-party evidenced.
Capacity bindings: exactly openai/codex (OpenAI entries) and zai/coding_plan (Z.ai entries). No additional live OpenAI scope, nothing inferred from window_id, no private bucket identifiers.

Task profiles (`model-policy.json`, policy_version 3 → 4)

schema_version stays 1. numeric_minima_included = true, numeric_minima_status = calibrated_m2c. Each of the eight formal profiles gains exactly one selector-facing calibrated_requirement (sole numeric definition; descriptive fields stay metadata).

Profile expansion mechanism (`selection_types.py`)

TaskProfileDefinition — safe profile ID + typed stored requirement; no model/provider/class affinity fields; pure to_requirement().
TaskProfileCatalog — unique IDs, canonical (sorted) deterministic serialization, unknown profile → SelectionContractValidationError. resolve(profile_id) is the only production profile resolver: pure, filesystem-independent, no capability inference, no model lookup, no scarcity, no explicit-override merge.

Tests (`tests/test_model_calibration.py`)

Catalog round-trip/determinism, exact rating/hard-property/binding matrices, provenance completeness, no-override, capability-only eligible-set scenarios (mechanical/routine_coding/general_reasoning: all four; deep_coding: Sol+GLM-5.3; scientific_review/translation: Sol; editorial/orchestration: Luna+Sol), exact requirement expansion, typed failure paths, policy calibration status. M2b contract tests remain intact except the two assertions of the now-superseded deferral state (numeric_minima deferred, policy_version 3), updated per D-025.

Governance

docs/model-calibration.md: rubric, tables, evidence hierarchy, uncertainty, update governance.
docs/decisions.md: D-025 recorded; U-006 RESOLVED by M2c/D-025; U-007 remains open (scarcity parameters → M2d).
Roadmap: M2c complete, next slice M2d. M2 PASS is not claimed.

Evidence pass (one bounded pass, 2026-09-06)

First-party OpenAI (2/3 pages), first-party Z.ai (4/4), Artificial Analysis (1 fetched + AA-published index values). No material contradiction with the accepted calibration was found.

Date deviation note (for reviewer attention): first-party evidence dates the GPT-5.6 family GA to 2026-07-09 (Sol preview 2026-06-26), not the expected 2026-07-29; the catalog records the evidenced GA date and documents the preview/GA distinction. GLM-5.3 is recorded at the announcement date 2026-08-14 (Z.ai's release-notes entry is labeled 2026-08-18); Flash 2026-08-26 matches exactly.

Non-scope (verified unchanged)

scarcity_router/capacity.py, scarcity_router/providers/*, scarcity_router/status.py untouched. No selector, ranking, scarcity, reservations, AA client, REST/MCP, additional providers or models.

Validation

make check — exit 0 (full unittest suite, 570 tests, 22 focused M2c calibration tests; basedpyright 0 errors / 0 warnings / 0 notes)
uv run basedpyright — 0 errors, 0 warnings, 0 notes
uv run python -m scarcity_router --help / status --help — OK
git diff --check origin/main — clean; no merge commits; origin/main..HEAD contains only this task's diff

Closes #36 ## Summary M2c populates the first accepted **content** for the frozen M2a/M2b contracts (D-025, resolving U-006). It does **not** implement scarcity, reservations, capacity aggregation or selection. ### Catalog artifact (`model-catalog.json`, version 1) Exactly four routing identities, generated and round-tripped through the M2b types (`ModelCatalog.from_dict`), serialized deterministically in `(provider, model, variant)` order: | Identity | Display name | Version date | | --- | --- | --- | | `openai/gpt-5.6-luna/max` | GPT-5.6 Luna Max | 2026-07-09 | | `openai/gpt-5.6-sol/high` | GPT-5.6 Sol High | 2026-07-09 | | `zai/glm-5.3/max` | GLM-5.3 Max | 2026-08-14 | | `zai/glm-5.3-flash/max` | GLM-5.3-Flash Max | 2026-08-26 | - Curated six-dimensional ratings on the frozen 1..5 scale (Luna 4/4/3/5/5/4, Sol 5/5/5/5/5/5, GLM-5.3 5/5/4/4/5/4, Flash 4/4/3/4/5/4) — the accepted calibration, not autonomously modified. - Every known rating carries ≥1 `EvidenceRef`, coarse confidence, `assessed_on = 2026-09-06` and a rationale. **No `HumanOverride`** is used for initial curation. - Tri-state hard properties (evidenced, first-party): GLM-5.3 `output_tokens = 128000` and `supports_vision = false` (explicit documented text-only inputs — a known negative serialized explicitly, not an omitted unknown); GLM-5.3-Flash `output_tokens = 128000` with vision true; Flash reasoning mode (`reasoning_effort: max`, always-on thinking) is first-party evidenced. - Capacity bindings: exactly `openai/codex` (OpenAI entries) and `zai/coding_plan` (Z.ai entries). No additional live OpenAI scope, nothing inferred from `window_id`, no private bucket identifiers. ### Task profiles (`model-policy.json`, policy_version 3 → 4) `schema_version` stays 1. `numeric_minima_included = true`, `numeric_minima_status = calibrated_m2c`. Each of the eight formal profiles gains exactly one selector-facing `calibrated_requirement` (sole numeric definition; descriptive fields stay metadata). ### Profile expansion mechanism (`selection_types.py`) - `TaskProfileDefinition` — safe profile ID + typed stored requirement; no model/provider/class affinity fields; pure `to_requirement()`. - `TaskProfileCatalog` — unique IDs, canonical (sorted) deterministic serialization, unknown profile → `SelectionContractValidationError`. `resolve(profile_id)` is the only production profile resolver: pure, filesystem-independent, no capability inference, no model lookup, no scarcity, no explicit-override merge. ### Tests (`tests/test_model_calibration.py`) Catalog round-trip/determinism, exact rating/hard-property/binding matrices, provenance completeness, no-override, capability-only eligible-set scenarios (mechanical/routine_coding/general_reasoning: all four; deep_coding: Sol+GLM-5.3; scientific_review/translation: Sol; editorial/orchestration: Luna+Sol), exact requirement expansion, typed failure paths, policy calibration status. M2b contract tests remain intact except the two assertions of the now-superseded deferral state (`numeric_minima` deferred, policy_version 3), updated per D-025. ### Governance - `docs/model-calibration.md`: rubric, tables, evidence hierarchy, uncertainty, update governance. - `docs/decisions.md`: **D-025** recorded; **U-006 RESOLVED** by M2c/D-025; **U-007 remains open** (scarcity parameters → M2d). - Roadmap: M2c complete, next slice M2d. **M2 PASS is not claimed.** ### Evidence pass (one bounded pass, 2026-09-06) First-party OpenAI (2/3 pages), first-party Z.ai (4/4), Artificial Analysis (1 fetched + AA-published index values). No material contradiction with the accepted calibration was found. **Date deviation note (for reviewer attention):** first-party evidence dates the GPT-5.6 family GA to **2026-07-09** (Sol preview 2026-06-26), not the expected 2026-07-29; the catalog records the evidenced GA date and documents the preview/GA distinction. GLM-5.3 is recorded at the announcement date 2026-08-14 (Z.ai's release-notes entry is labeled 2026-08-18); Flash 2026-08-26 matches exactly. ### Non-scope (verified unchanged) `scarcity_router/capacity.py`, `scarcity_router/providers/*`, `scarcity_router/status.py` untouched. No selector, ranking, scarcity, reservations, AA client, REST/MCP, additional providers or models. ## Validation - `make check` — exit 0 (full unittest suite, 570 tests, 22 focused M2c calibration tests; basedpyright 0 errors / 0 warnings / 0 notes) - `uv run basedpyright` — 0 errors, 0 warnings, 0 notes - `uv run python -m scarcity_router --help` / `status --help` — OK - `git diff --check origin/main` — clean; no merge commits; `origin/main..HEAD` contains only this task's diff