Publish official Terminal-Bench 4.0 coding evidence #48

Merged

kilo merged 1 commit from issue-47-terminal-bench-evidence-isolated into develop

2026-10-28 12:52:57 +00:00

kilo commented

2026-10-28 12:43:20 +00:00

Member

Refs #47

Summary

Adds a separately scoped Terminal-Bench 4.0 adapter using only four maintainer-published leaderboard/submissions/*.json records pinned to harbor-framework/terminal-bench@6964e9252c90227fc1cfa26cbc59aee4134792e4.
Records resolution rate in native percent units, official per-task 95% CI method, selected trials/successes, exact model route/display identity, effort, agent/version, model release date, dataset digest, source job and source revision. Evaluation end time/model revisions remain UNKNOWN; the GLM-5.3 anthropic/glm-5.3 source alias is preserved beside its Z.ai model metadata.
Composes the new evidence with the existing M1-01 DeepSWE comparison without changing M1-01/Z.ai publication scopes or /1//2 contracts. Comparisons remain configuration-specific; no score normalization, Router rating, winner or admission rule is added.
Includes source-linked token/cost telemetry in the read-only receipt only; no subscription/Kilo cost inference or raw Hub trial/trajectory data is copied.
Rights basis: selected JSON entries are contributions in the exact-pinned Apache-2.0 repository under Sections 2/5 and the documented submission workflow. No task dataset is copied; no upstream mutation is made.

Reproducible receipt

CODING_BENCHMARK_COMPARISON.md records the real source-to-publication HTTP 200 capture, exact file byte counts/SHA-256, separate immutable publication digest, candidate comparison, licenses, CI method and source limitations.

Validation

make check: 213 tests, Ruff, basedpyright and diff checks pass.
Offline source distribution/wheel and installed legacy/current/producer/downgrade tests pass, including the new acquisition/Terminal-Bench suites.

No Router/Kernel/Console changes, external credentials or raw Harbor trial payloads are included.

Refs #47 ## Summary - Adds a separately scoped Terminal-Bench 4.0 adapter using only four maintainer-published `leaderboard/submissions/*.json` records pinned to `harbor-framework/terminal-bench@6964e9252c90227fc1cfa26cbc59aee4134792e4`. - Records resolution rate in native percent units, official per-task 95% CI method, selected trials/successes, exact model route/display identity, effort, agent/version, model release date, dataset digest, source job and source revision. Evaluation end time/model revisions remain UNKNOWN; the GLM-5.3 `anthropic/glm-5.3` source alias is preserved beside its Z.ai model metadata. - Composes the new evidence with the existing M1-01 DeepSWE comparison without changing M1-01/Z.ai publication scopes or `/1`/`/2` contracts. Comparisons remain configuration-specific; no score normalization, Router rating, winner or admission rule is added. - Includes source-linked token/cost telemetry in the read-only receipt only; no subscription/Kilo cost inference or raw Hub trial/trajectory data is copied. - Rights basis: selected JSON entries are contributions in the exact-pinned Apache-2.0 repository under Sections 2/5 and the documented submission workflow. No task dataset is copied; no upstream mutation is made. ## Reproducible receipt `CODING_BENCHMARK_COMPARISON.md` records the real source-to-publication HTTP 200 capture, exact file byte counts/SHA-256, separate immutable publication digest, candidate comparison, licenses, CI method and source limitations. ## Validation - `make check`: 213 tests, Ruff, basedpyright and diff checks pass. - Offline source distribution/wheel and installed legacy/current/producer/downgrade tests pass, including the new acquisition/Terminal-Bench suites. No Router/Kernel/Console changes, external credentials or raw Harbor trial payloads are included.

kilo added 1 commit

2026-10-28 12:43:20 +00:00