Flaky under load: test_scenario_11 client-disconnect cancel can assert before the cancel reaches the worker reader

kilo commented

2026-09-26 23:57:54 +00:00

Member

Discovered during program #132 full-gate runs on compat/logical-model-resolution (child B, #134); the failing path is untouched by that branch's diff, and the class passes 8/8 in isolation on BOTH the branch and pristine develop @ 893d2cb — evidence points to a pre-existing, load-sensitive race.

Evidence (2026-09-26 session)

Full-suite make check: 2 failures of ~7 runs this session, always
test_e2e_execution.CancellationTests.test_scenario_11_client_disconnect_propagates_cancel
— AssertionError: [] is not true : cancel never reached the worker (tests/test_e2e_execution.py:875).
Isolation loop of the class: 8/8 OK on the branch, 8/8 OK on pristine develop — the race only manifests under full-suite system load.

Root-cause hypothesis (not verified)

In test_scenario_11_client_disconnect_propagates_cancel, after the client half-closes (SHUT_WR) the test drains the response socket:

while time.monotonic() < deadline and not worker.cancels:
    piece = raw.recv(4096)  # drain; may EOF or time out
    if not piece:
        break  # the server closed its side after cancelling

Under load the server's FIN can be observed (EOF → break) before the cancel message has been read and recorded by the worker's reader thread (worker.cancels is populated by start_reader()), so the assertion races cancel delivery against FIN observation. The 15 s deadline never gets a chance to protect the assertion because a clean EOF exits the loop early.

Fix direction (root cause, no retries, no timeout bumps)

Reconcile the two exit conditions: on EOF, keep polling worker.cancels for a short bounded grace window (the cancel is expected to be in flight at that point), or wait on the worker reader's cancel-received event instead of inferring from socket EOF. The wait must remain deterministic and bounded.

Non-goals

No product behavior change is implied by the evidence; do not weaken the scenario's assertion (cancel propagation to the exact owning worker is the property under test).

Discovered during program #132 full-gate runs on `compat/logical-model-resolution` (child B, #134); the failing path is untouched by that branch's diff, and the class passes 8/8 in isolation on BOTH the branch and pristine `develop` @ `893d2cb` — evidence points to a pre-existing, load-sensitive race. ## Evidence (2026-09-26 session) - Full-suite `make check`: 2 failures of ~7 runs this session, always `test_e2e_execution.CancellationTests.test_scenario_11_client_disconnect_propagates_cancel` — `AssertionError: [] is not true : cancel never reached the worker` (tests/test_e2e_execution.py:875). - Isolation loop of the class: 8/8 OK on the branch, 8/8 OK on pristine `develop` — the race only manifests under full-suite system load. ## Root-cause hypothesis (not verified) In `test_scenario_11_client_disconnect_propagates_cancel`, after the client half-closes (`SHUT_WR`) the test drains the response socket: ```python while time.monotonic() < deadline and not worker.cancels: piece = raw.recv(4096) # drain; may EOF or time out if not piece: break # the server closed its side after cancelling ``` Under load the server's FIN can be observed (EOF → `break`) before the cancel message has been read and recorded by the worker's reader thread (`worker.cancels` is populated by `start_reader()`), so the assertion races cancel *delivery* against FIN *observation*. The 15 s deadline never gets a chance to protect the assertion because a clean EOF exits the loop early. ## Fix direction (root cause, no retries, no timeout bumps) Reconcile the two exit conditions: on EOF, keep polling `worker.cancels` for a short bounded grace window (the cancel is expected to be in flight at that point), or wait on the worker reader's cancel-received event instead of inferring from socket EOF. The wait must remain deterministic and bounded. ## Non-goals No product behavior change is implied by the evidence; do not weaken the scenario's assertion (cancel propagation to the exact owning worker is the property under test).

adrian.tkacz referenced this issue from a commit

2026-09-26 23:58:15 +00:00

feat: logical model resolution + adoption-aware discovery (D-055, #134)

kilo referenced this issue

2026-09-26 23:58:41 +00:00

Logical OpenAI model discovery + bare-model resolution (D-055, #134) #144

kilo referenced this issue

2026-09-27 00:38:26 +00:00

Bounded reasoning dialect normalization (#135) #145

kilo referenced this issue

2026-09-27 00:47:15 +00:00

Bounded reasoning dialect normalization (#135) #145

kilo referenced this issue

2026-09-27 16:35:19 +00:00

Harness-independent execution backend architecture (D-056, #146) #147

kilo referenced this issue

2026-09-27 18:19:40 +00:00

Harness-independent execution backend architecture (D-056, #146) #147

kilo referenced this issue

2026-09-27 19:06:01 +00:00

Harness-independent execution backend architecture (D-056, #146) #147

kilo referenced this issue

2026-09-27 21:01:22 +00:00

Representative-client compatibility + operations hardening program (ZCode acceptance follow-up) #132

kilo referenced this issue

2026-09-28 00:08:40 +00:00

Public contract: serialized candidates carry their catalog reasoning effort (D-057, #148) #149

kilo commented

2026-09-28 00:14:49 +00:00

Author

Member

Second variant of the same load-sensitive race family, observed 2026-09-28 during the D-057 / PR #149 full-gate runs on contract/reasoning-effort-serialization (head 2ffc224, based on develop @ f574e51):

Full-suite make check: 1 failure of 2 runs this session — test_e2e_execution.CancellationTests.test_scenario_11_cancel_reaches_exactly_the_owning_worker (tests/test_e2e_execution.py:906): AssertionError: [] is not true (worker.cancels empty — cancel not yet recorded when asserted).
Isolation loop: 3/3 OK on the branch, 8/8 OK on pristine develop @ f574e51 — same shape as the originally reported scenario: the assertion races cancel delivery against an earlier exit condition, manifesting only under full-suite system load. The failing path is untouched by the #149 diff (candidate-serialization change only).
Recorded here as additional evidence for the root-cause fix direction in the issue body (reconcile the two exit conditions on the worker-reader wait); no product behavior change implied.

Second variant of the same load-sensitive race family, observed 2026-09-28 during the D-057 / PR #149 full-gate runs on `contract/reasoning-effort-serialization` (head `2ffc224`, based on `develop` @ `f574e51`): - Full-suite `make check`: 1 failure of 2 runs this session — `test_e2e_execution.CancellationTests.test_scenario_11_cancel_reaches_exactly_the_owning_worker` (tests/test_e2e_execution.py:906): `AssertionError: [] is not true` (`worker.cancels` empty — cancel not yet recorded when asserted). - Isolation loop: 3/3 OK on the branch, 8/8 OK on pristine `develop` @ `f574e51` — same shape as the originally reported scenario: the assertion races cancel delivery against an earlier exit condition, manifesting only under full-suite system load. The failing path is untouched by the #149 diff (candidate-serialization change only). - Recorded here as additional evidence for the root-cause fix direction in the issue body (reconcile the two exit conditions on the worker-reader wait); no product behavior change implied.

adrian.tkacz referenced this issue from a commit

2026-09-28 00:31:32 +00:00

feat: gateway effective limits + Codex output-token semantics (D-057, #136)

kilo referenced this issue

2026-09-28 00:31:54 +00:00

Gateway effective limits + Codex output-token semantics (D-058, #136) #152

adrian.tkacz referenced this issue from a commit

2026-09-28 07:54:12 +00:00

feat: gateway effective limits + Codex output-token semantics (D-058, #136)

kilo referenced this issue

2026-09-28 11:10:39 +00:00

Gateway effective limits + Codex output-token semantics (D-058, #136) #152