M1-01: Publish source-backed coding benchmark evidence for Sol, Luna and GLM #45

New issue

Closed

opened 2026-10-27 22:09:12 +00:00 by kilo · 1 comment

kilo commented

2026-10-27 22:09:12 +00:00

Member

Goal

Deliver a small, real-source, rights-verified MI evidence slice useful for assessing GPT-6.1 Sol High, GPT-6 Luna Max, GLM-5.3-Flash and, where relevant, GLM-5.3 for autonomous coding work. Preserve MI's accepted ADAPT architecture: MI publishes contextual public evidence; Router owns calibration, model selection and private execution policy.

Acceptance criteria

Perform exactly one bounded authenticated Artificial Analysis Free-endpoint probe using the explicitly supplied temporary credential, reporting only safe metadata (status/tier/rate limits/schema/pagination/candidate availability/field categories); at most one bounded Pro list probe only if observed tier supports it. No credential or authenticated response is retained or published. Treat AA product-use rights as UNVERIFIED unless independently verified; publish no AA scores absent express permission.
Verify a public benchmark source's authority, provenance and actual rights for intended reuse/distribution. A software repository license does not automatically license leaderboard data. Continue with another source such as selected models.dev records when appropriate; explicitly exclude unverified data.
Implement only the smallest compatible source acquisition/normalization extension using existing Benchmark/ModelClaim/Observation/PublicEvidence and immutable publication contracts, with bounds, complete-source validation, source references and retained history. Preserve existing Z.ai scope and /1//2 semantics.
Provide reproducible read-only evidence comparison for Sol High, Luna Max, GLM-5.3-Flash and relevant GLM-5.3, preserving exact tested variants/configuration, harness, metric/unit, provenance, uncertainty and non-comparability. Mark unknowns explicitly; no fabricated dates, score, winner, Router rating or cost inference.
Cover credential/probe safety, source rights, malformed/oversized/changed source, identity/configuration/metric/unit/unknowns, comparability, history, source disappearance/retraction, existing scope and operator/publication compatibility with deterministic synthetic tests. Provide a reproducible source-to-publication receipt for any real source claimed.
Run repository make check, packaging/build and applicable installed-distribution tests; open one issue-linked PR to develop, obtain fresh independent whole-PR approval, and integrate only via authorized /loop merge/currentness/acceptance gates.

Boundaries

No Router/Kernel/Console changes, no AA score redistribution without verified permission, no fabricated benchmark evidence, no ranking/admission algorithm, paid access, quota exhaustion, inference, deployment or release. Keep this issue open until integrated acceptance is verified.

## Goal Deliver a small, real-source, rights-verified MI evidence slice useful for assessing GPT-6.1 Sol High, GPT-6 Luna Max, GLM-5.3-Flash and, where relevant, GLM-5.3 for autonomous coding work. Preserve MI's accepted ADAPT architecture: MI publishes contextual public evidence; Router owns calibration, model selection and private execution policy. ## Acceptance criteria 1. Perform exactly one bounded authenticated Artificial Analysis Free-endpoint probe using the explicitly supplied temporary credential, reporting only safe metadata (status/tier/rate limits/schema/pagination/candidate availability/field categories); at most one bounded Pro list probe only if observed tier supports it. No credential or authenticated response is retained or published. Treat AA product-use rights as UNVERIFIED unless independently verified; publish no AA scores absent express permission. 2. Verify a public benchmark source's authority, provenance and actual rights for intended reuse/distribution. A software repository license does not automatically license leaderboard data. Continue with another source such as selected models.dev records when appropriate; explicitly exclude unverified data. 3. Implement only the smallest compatible source acquisition/normalization extension using existing Benchmark/ModelClaim/Observation/PublicEvidence and immutable publication contracts, with bounds, complete-source validation, source references and retained history. Preserve existing Z.ai scope and `/1`/`/2` semantics. 4. Provide reproducible read-only evidence comparison for Sol High, Luna Max, GLM-5.3-Flash and relevant GLM-5.3, preserving exact tested variants/configuration, harness, metric/unit, provenance, uncertainty and non-comparability. Mark unknowns explicitly; no fabricated dates, score, winner, Router rating or cost inference. 5. Cover credential/probe safety, source rights, malformed/oversized/changed source, identity/configuration/metric/unit/unknowns, comparability, history, source disappearance/retraction, existing scope and operator/publication compatibility with deterministic synthetic tests. Provide a reproducible source-to-publication receipt for any real source claimed. 6. Run repository `make check`, packaging/build and applicable installed-distribution tests; open one issue-linked PR to develop, obtain fresh independent whole-PR approval, and integrate only via authorized `/loop` merge/currentness/acceptance gates. ## Boundaries No Router/Kernel/Console changes, no AA score redistribution without verified permission, no fabricated benchmark evidence, no ranking/admission algorithm, paid access, quota exhaustion, inference, deployment or release. Keep this issue open until integrated acceptance is verified.

kilo referenced this issue

2026-10-27 22:38:44 +00:00

Publish source-backed coding benchmark evidence #46

kilo commented

2026-10-27 22:52:16 +00:00

Author

Member

M1-01 implementation integrated through PR #46: #46

Merge commit: eb328882a821533ba7f8eeee3706cdd37c7b8e03 on develop.

The pinned, MIT-licensed models.dev evidence includes DeepSWE v1.1 reports for Luna Max (66.6, metric score), GLM-5.3 (66.9, metric resolved) and GLM-5.3-Flash (63.4, metric resolved), plus a provider-reported GLM-5.3 SWE-Marathon v1.1 result (42.5, metric score) with modified checks. Native units remain UNKNOWN. Z.ai's pinned cards establish the GLM harness/configuration; Sol 6.1 has no benchmark score in this licensed source slice. The comparison preserves these limits and infers no winner or Router rating. The official SWE-Marathon leaderboard dataset and all AA scores are excluded.

A single bounded AA Free probe succeeded but exposed no individual benchmark scores; AA_PRODUCT_USE_RIGHTS=UNVERIFIED. No credential or raw authenticated response was retained or published.

Validation: make check passed (198 tests, Ruff, basedpyright and diff checks); offline sdist/wheel and installed legacy/current/producer/downgrade plus new adapter/probe tests passed. No Router/Kernel/Console changes; integration does not imply product GO.

M1-01 implementation integrated through PR #46: https://forgejo.creatidy.com/Creatidy/model-intelligence/pulls/46 Merge commit: `eb328882a821533ba7f8eeee3706cdd37c7b8e03` on `develop`. The pinned, MIT-licensed models.dev evidence includes DeepSWE v1.1 reports for Luna Max (66.6, metric `score`), GLM-5.3 (66.9, metric `resolved`) and GLM-5.3-Flash (63.4, metric `resolved`), plus a provider-reported GLM-5.3 SWE-Marathon v1.1 result (42.5, metric `score`) with modified checks. Native units remain UNKNOWN. Z.ai's pinned cards establish the GLM harness/configuration; Sol 6.1 has no benchmark score in this licensed source slice. The comparison preserves these limits and infers no winner or Router rating. The official SWE-Marathon leaderboard dataset and all AA scores are excluded. A single bounded AA Free probe succeeded but exposed no individual benchmark scores; `AA_PRODUCT_USE_RIGHTS=UNVERIFIED`. No credential or raw authenticated response was retained or published. Validation: `make check` passed (198 tests, Ruff, basedpyright and diff checks); offline sdist/wheel and installed legacy/current/producer/downgrade plus new adapter/probe tests passed. No Router/Kernel/Console changes; integration does not imply product GO.

kilo closed this issue

2026-10-27 22:52:20 +00:00