M1-02: Source-backed Terminal-Bench 4.0 evidence for Sol, Luna and GLM #47

New issue

Closed

opened 2026-10-28 11:17:33 +00:00 by kilo · 1 comment

kilo commented

2026-10-28 11:17:33 +00:00

Member

Goal

Deliver a real, rights-verified, source-backed Terminal-Bench 4.0 evidence comparison for GPT-6.1 Sol, GPT-6 Luna, GLM-5.3 and GLM-5.3-Flash as measured model-agent configurations. Extend M1-01's separate benchmark evidence scope without changing MI's ADAPT architecture or existing /1, /2, Z.ai or DeepSWE history.

Acceptance criteria

Independently inspect the current official Terminal-Bench 4.0 leaderboard, benchmark repository, maintained leaderboard/submission infrastructure and actual source revisions. Establish dataset/version/task set, organizer/evaluator, exact model identity and effort, agent/harness/version, native resolution-rate metric/unit/direction, confidence interval/method, tasks/trials/aggregation, evaluation date versus model release date, tokens/cost basis, source authority and provenance.
Verify rights separately for benchmark code, tasks/data, leaderboard observations and submission artifacts. Do not assume Apache-2.0 source code licenses independently published result data or downstream redistribution. Publish only selected observations whose intended MI distribution is supported; retain an exact UNVERIFIED right for anything not supported and do not fabricate positive publication acceptance.
Reuse model_intelligence.benchmarks and its existing typed claims/acquisition/immutable publication. Implement only a bounded, separately scoped source extension; strict allowlists, byte limits, full-generation validation, exact identities/configurations, original units, confidence intervals and source references. Preserve history and source absence versus explicit retraction. Do not build another catalog/crawler/service/database/scheduler or schema family.
Extend the read-only comparison with the four target configurations: Sol max/Codex; Luna max/Codex; GLM-5.3 max/Claude Code; GLM-5.3-Flash none/Claude Code, after verifying all values against source. Do not equate max with Sol High/Luna Max in Kilo, transfer between Sol and Sol 6.1 or between GLM variants, or compare different dataset versions/leaderboard implementations as controlled trials. Mark missing evidence UNKNOWN; no score normalization, rating, winner, quota/subscription cost inference or automatic routing.
Add deterministic synthetic regression tests for identity/variant, metric/unit/range, confidence bounds, source/revision changes, unknown dates, incomplete/malformed/oversized acquisition, rights refusals, retained history, source disappearance/retraction, incomparable configurations, no rating conversion and M1-01/Z.ai compatibility. Include a reproducible real source-to-publication receipt for each positive published source.
Run make check, source distribution/wheel/installed compatibility checks. Open one issue-linked PR, obtain a fresh independent whole-PR approval, integrate through the authorized /loop merge gates, and close only after integrated acceptance is verified.

Boundaries

No Router, Kernel, Console or other repository changes; no production activation, deployment, release or product GO. No unlicensed dataset redistribution, fabricated score/configuration/date, leaderboard aggregation of incomparable submissions or automatic selector/ranker.

## Goal Deliver a real, rights-verified, source-backed Terminal-Bench 4.0 evidence comparison for GPT-6.1 Sol, GPT-6 Luna, GLM-5.3 and GLM-5.3-Flash as measured model-agent configurations. Extend M1-01's separate benchmark evidence scope without changing MI's ADAPT architecture or existing `/1`, `/2`, Z.ai or DeepSWE history. ## Acceptance criteria 1. Independently inspect the current official Terminal-Bench 4.0 leaderboard, benchmark repository, maintained leaderboard/submission infrastructure and actual source revisions. Establish dataset/version/task set, organizer/evaluator, exact model identity and effort, agent/harness/version, native resolution-rate metric/unit/direction, confidence interval/method, tasks/trials/aggregation, evaluation date versus model release date, tokens/cost basis, source authority and provenance. 2. Verify rights separately for benchmark code, tasks/data, leaderboard observations and submission artifacts. Do not assume Apache-2.0 source code licenses independently published result data or downstream redistribution. Publish only selected observations whose intended MI distribution is supported; retain an exact `UNVERIFIED` right for anything not supported and do not fabricate positive publication acceptance. 3. Reuse `model_intelligence.benchmarks` and its existing typed claims/acquisition/immutable publication. Implement only a bounded, separately scoped source extension; strict allowlists, byte limits, full-generation validation, exact identities/configurations, original units, confidence intervals and source references. Preserve history and source absence versus explicit retraction. Do not build another catalog/crawler/service/database/scheduler or schema family. 4. Extend the read-only comparison with the four target configurations: Sol max/Codex; Luna max/Codex; GLM-5.3 max/Claude Code; GLM-5.3-Flash none/Claude Code, after verifying all values against source. Do not equate max with Sol High/Luna Max in Kilo, transfer between Sol and Sol 6.1 or between GLM variants, or compare different dataset versions/leaderboard implementations as controlled trials. Mark missing evidence UNKNOWN; no score normalization, rating, winner, quota/subscription cost inference or automatic routing. 5. Add deterministic synthetic regression tests for identity/variant, metric/unit/range, confidence bounds, source/revision changes, unknown dates, incomplete/malformed/oversized acquisition, rights refusals, retained history, source disappearance/retraction, incomparable configurations, no rating conversion and M1-01/Z.ai compatibility. Include a reproducible real source-to-publication receipt for each positive published source. 6. Run `make check`, source distribution/wheel/installed compatibility checks. Open one issue-linked PR, obtain a fresh independent whole-PR approval, integrate through the authorized `/loop` merge gates, and close only after integrated acceptance is verified. ## Boundaries No Router, Kernel, Console or other repository changes; no production activation, deployment, release or product GO. No unlicensed dataset redistribution, fabricated score/configuration/date, leaderboard aggregation of incomparable submissions or automatic selector/ranker.

kilo referenced this issue

2026-10-28 12:43:20 +00:00

Publish official Terminal-Bench 4.0 coding evidence #48

kilo commented

2026-10-28 12:54:00 +00:00

Author

Member

M1-02 is integrated through PR #48: #48

Merge commit e8a6b81546994438038d22fd08fc041c6901d9cd is on develop; it preserves the reviewed base eb328882a821533ba7f8eeee3706cdd37c7b8e03 and approved head 25c068d08b86f920aeee5d69c8cff40abadac8ef.

The separate immutable Terminal-Bench 4.0.0 publication uses four Apache-2.0-contributed summary records from harbor-framework/terminal-bench@6964e9252c90227fc1cfa26cbc59aee4134792e; dataset content is pinned to sha256:39d9f44b40420cde8fdcc087579c0d72a7e14fa3656d603c3f0d22fb35e27732. It records Sol max/Codex 0.159.0 at 58.18% ± 3.11pp (192/330), Luna max/Codex 0.159.0 at 16.36% ± 2.72pp (54/330), GLM-5.3 max/Claude Code 2.1.207 at 41.82% ± 3.23pp (138/330), and GLM-5.3-Flash none/Claude Code 2.1.285 at 35.76% ± 3.54pp (118/330). The source CI is the official per-task standard-error method with z=1.96; it is not a pairwise significance test. The release dates, source-job starts, model-revision unknowns, GLM source-ID/Z.ai metadata mismatch, and 330 selected trials are documented separately.

Sol/Luna share the submitted Codex version/effort, but the result is not a Kilo Sol High run. GLM submissions differ in effort/agent version and are not controlled model-only comparisons. No score/rating conversion, winner or automatic routing is inferred. Source-reported tokens and cost are contextual job telemetry only, not subscription, quota or Kilo-task cost. Only the four licensed summary JSONs are used; no task dataset, trial-ID lists, raw Hub payloads or trajectories are copied.

Validation: make check passed (213 tests, Ruff, basedpyright and diff checks); offline sdist/wheel and installed legacy/current/producer/downgrade plus new Terminal-Bench/acquisition tests passed. Independent review #2 approved the exact head/base. Integration does not imply product GO or Kilo-specific validation.

M1-02 is integrated through PR #48: https://forgejo.creatidy.com/Creatidy/model-intelligence/pulls/48 Merge commit `e8a6b81546994438038d22fd08fc041c6901d9cd` is on `develop`; it preserves the reviewed base `eb328882a821533ba7f8eeee3706cdd37c7b8e03` and approved head `25c068d08b86f920aeee5d69c8cff40abadac8ef`. The separate immutable Terminal-Bench 4.0.0 publication uses four Apache-2.0-contributed summary records from `harbor-framework/terminal-bench@6964e9252c90227fc1cfa26cbc59aee4134792e`; dataset content is pinned to `sha256:39d9f44b40420cde8fdcc087579c0d72a7e14fa3656d603c3f0d22fb35e27732`. It records Sol max/Codex 0.159.0 at 58.18% ± 3.11pp (192/330), Luna max/Codex 0.159.0 at 16.36% ± 2.72pp (54/330), GLM-5.3 max/Claude Code 2.1.207 at 41.82% ± 3.23pp (138/330), and GLM-5.3-Flash none/Claude Code 2.1.285 at 35.76% ± 3.54pp (118/330). The source CI is the official per-task standard-error method with z=1.96; it is not a pairwise significance test. The release dates, source-job starts, model-revision unknowns, GLM source-ID/Z.ai metadata mismatch, and 330 selected trials are documented separately. Sol/Luna share the submitted Codex version/effort, but the result is not a Kilo Sol High run. GLM submissions differ in effort/agent version and are not controlled model-only comparisons. No score/rating conversion, winner or automatic routing is inferred. Source-reported tokens and cost are contextual job telemetry only, not subscription, quota or Kilo-task cost. Only the four licensed summary JSONs are used; no task dataset, trial-ID lists, raw Hub payloads or trajectories are copied. Validation: `make check` passed (213 tests, Ruff, basedpyright and diff checks); offline sdist/wheel and installed legacy/current/producer/downgrade plus new Terminal-Bench/acquisition tests passed. Independent review #2 approved the exact head/base. Integration does not imply product GO or Kilo-specific validation.

kilo closed this issue

2026-10-28 12:54:20 +00:00