Publish source-backed coding benchmark evidence #46

Merged

kilo merged 1 commit from issue-45-coding-benchmark-evidence into develop

2026-10-27 22:51:40 +00:00

kilo commented

2026-10-27 22:38:44 +00:00

Member

Refs #45

Summary

Adds a separately scoped, bounded models.dev adapter for pinned MIT-licensed model TOMLs, using MI's existing immutable evidence/publication contract and preserving history.
Adds a read-only, unranked comparison for GPT-6.1 Sol, GPT-6 Luna, GLM-5.3-Flash and GLM-5.3. It retains source metric labels/configurations and marks missing units, dates, harnesses and scores UNKNOWN; it does not infer a winner or Router rating.
Records pinned Z.ai provider-run configurations for the GLM DeepSWE and SWE-Marathon observations. The GLM SWE-Marathon report notes modified checks and is not represented as the organizer's leaderboard row. No unlicensed official SWE-Marathon leaderboard scores are copied.
Adds a one-request metadata-only AA Free endpoint diagnostic. The private probe result is not published or retained; no AA score is included (AA_PRODUCT_USE_RIGHTS=UNVERIFIED).
Leaves Z.ai publication scope and other repositories unchanged.

Evidence

The reproducible source-to-publication receipt and candidate comparison are in CODING_BENCHMARK_COMPARISON.md. Source set: models.dev commit 3fb4bb98e633ff1af1b94f84ba07c8549e859aa8 (exact-revision MIT); provider configuration references are pinned in the adapter.

Validation

make check: locked sync, Ruff, basedpyright, 198 tests and both diff checks passed.
Offline tests/verify_distribution.py: source distribution, wheels, and installed legacy/current/producer/downgrade checks passed; installed new adapter/probe tests passed.

No credential or authenticated AA response is present in the branch.

Refs #45 ## Summary - Adds a separately scoped, bounded models.dev adapter for pinned MIT-licensed model TOMLs, using MI's existing immutable evidence/publication contract and preserving history. - Adds a read-only, unranked comparison for GPT-6.1 Sol, GPT-6 Luna, GLM-5.3-Flash and GLM-5.3. It retains source metric labels/configurations and marks missing units, dates, harnesses and scores UNKNOWN; it does not infer a winner or Router rating. - Records pinned Z.ai provider-run configurations for the GLM DeepSWE and SWE-Marathon observations. The GLM SWE-Marathon report notes modified checks and is not represented as the organizer's leaderboard row. No unlicensed official SWE-Marathon leaderboard scores are copied. - Adds a one-request metadata-only AA Free endpoint diagnostic. The private probe result is not published or retained; no AA score is included (`AA_PRODUCT_USE_RIGHTS=UNVERIFIED`). - Leaves Z.ai publication scope and other repositories unchanged. ## Evidence The reproducible source-to-publication receipt and candidate comparison are in `CODING_BENCHMARK_COMPARISON.md`. Source set: models.dev commit `3fb4bb98e633ff1af1b94f84ba07c8549e859aa8` (exact-revision MIT); provider configuration references are pinned in the adapter. ## Validation - `make check`: locked sync, Ruff, basedpyright, 198 tests and both diff checks passed. - Offline `tests/verify_distribution.py`: source distribution, wheels, and installed legacy/current/producer/downgrade checks passed; installed new adapter/probe tests passed. No credential or authenticated AA response is present in the branch.

kilo added 1 commit

2026-10-27 22:38:45 +00:00