Test suite stabilization: fix 502 flake in packaged-worker Codex E2E, eliminate resource leaks, reduce runtime #111

New issue

Closed

opened 2026-09-22 16:43:39 +00:00 by kilo · 0 comments

kilo commented

2026-09-22 16:43:39 +00:00

Member

Summary

Forgejo CI run on develop (post-#109) reports:

Ran 1895 tests in 676.418s
FAILED (failures=1, skipped=2)

with the single failure in
PackagedWorkerPathTests.test_console_script_worker_discovers_codex_pairs_and_serves
(expected 200, received 502 backend_failure), plus many ResourceWarnings
(unclosed BufferedReader / socket / TextIOWrapper) and a suspiciously long
runtime (~676 s).

Scope

Reproduce and fix the 502 flake at root cause (no retries, no timeout bumps, no weakened assertions).
Audit and fix resource lifecycle ownership: subprocesses, pipes, sockets, TLS sockets, files, worker/Codex reader+drainer threads, fake app-server processes.
Zero project-owned ResourceWarnings at the end of a clean run; no orphan subprocesses/sockets/threads after teardown.
Replace timeout-as-synchronization with deterministic readiness signals where measured.
Add lightweight timing observability (e.g. SR_TEST_TIMING=1), no new dependencies, no dashboards.
Keep ONE normal gate: make check. No make acceptance, no make release-check, no new test framework (no pytest), no coverage removal.
Do not change provider/routing behavior unless the 502 proves a real product lifecycle bug.

Out of scope

main promotion, v0.1.0 tag/release/publish, live Codex login, Windows release acceptance.

Acceptance

Flaky test 20/20 consecutive passes; full Codex E2E file 5/5 passes.
Full gate green: make check, make package-check, make guardrails, git diff --check, basedpyright.
Baseline vs after timing report with top slowest tests/files and justification of remaining cost.
One independent GLM-5.3 (thinking=MAX) review; verdict recorded.
One PR fix/test-suite-stabilization -> develop, left open for human review.

## Summary Forgejo CI run on `develop` (post-#109) reports: ``` Ran 1895 tests in 676.418s FAILED (failures=1, skipped=2) ``` with the single failure in `PackagedWorkerPathTests.test_console_script_worker_discovers_codex_pairs_and_serves` (expected 200, received 502 backend_failure), plus many `ResourceWarning`s (unclosed BufferedReader / socket / TextIOWrapper) and a suspiciously long runtime (~676 s). ## Scope - Reproduce and fix the 502 flake at root cause (no retries, no timeout bumps, no weakened assertions). - Audit and fix resource lifecycle ownership: subprocesses, pipes, sockets, TLS sockets, files, worker/Codex reader+drainer threads, fake app-server processes. - Zero project-owned ResourceWarnings at the end of a clean run; no orphan subprocesses/sockets/threads after teardown. - Replace timeout-as-synchronization with deterministic readiness signals where measured. - Add lightweight timing observability (e.g. `SR_TEST_TIMING=1`), no new dependencies, no dashboards. - Keep ONE normal gate: `make check`. No `make acceptance`, no `make release-check`, no new test framework (no pytest), no coverage removal. - Do not change provider/routing behavior unless the 502 proves a real product lifecycle bug. ## Out of scope - `main` promotion, v0.1.0 tag/release/publish, live Codex login, Windows release acceptance. ## Acceptance - Flaky test 20/20 consecutive passes; full Codex E2E file 5/5 passes. - Full gate green: `make check`, `make package-check`, `make guardrails`, `git diff --check`, basedpyright. - Baseline vs after timing report with top slowest tests/files and justification of remaining cost. - One independent GLM-5.3 (thinking=MAX) review; verdict recorded. - One PR `fix/test-suite-stabilization` -> `develop`, left open for human review.