F22.3 — Behavioral tests and performance baselines
| State | Closed |
| Parent | E22 |
| Iteration | Release 9 |
| Start Date | 2026-11-14 |
| Closed Date | 2026-11-16 |
| Predecessors | — |
Stories: 3 Closed
| Story | State | Depends on |
|---|---|---|
| S22.3.1 — Keep contract tests independent of source layout | Closed (2026-11-14) | — |
| S22.3.2 — Measure representative work | Closed (2026-11-14) | — |
| S22.3.3 — Reduce repeated test setup where measured | Closed (2026-11-16) | S22.3.2 |
Protect the behavior being simplified and measure expensive work before changing it.
Context
The suite is mostly named by behavior already: only nine test names carry
story-like IDs, all in commons block tests. Root built_wheels builds all ten
packages once per scaffold session; scaffold environments resolve third-party
requirements from the index. SQLAlchemy upsert tests write 33,000 rows. Commons
task tests directly call _start, renaming the pytest main thread, and inject
an impossible orphan into private pool state. These are concrete test-coupling
cases; setup/large-input work remains a measurement target.
S22.3.1 — Keep contract tests independent of source layout
State: Closed (2026-11-14)
Inventory private patches, source-inspection assertions, duplicate test builders and time-sensitive assertions only in areas this epic changes. Rename the nine block tests by behavior. Preserve architecture checks that intentionally inspect imports or examples.
Acceptance:
- Every removed implementation assertion has equivalent public-behavior coverage or a documented reason it was only an architecture check.
- Collected behavioral cases are preserved, including parameter variants, negative/security cases and optional-extra installs.
- Task execution tests use submit/futures/shutdown and preserve the test runner thread name; timing tests use synchronization or controllable clocks where possible, without blanket deletion of sleeps or AST checks.
- The orphan-future shutdown test is deleted;
TaskPool.shutdown'sfuture is Nonebranch stays for F22.4. Themap_settledcancellation test keeps its private queue poll, because no public API exposes queued state.
Outcome: commons collects 1,057 cases, down from 1,058: only the orphan test
is gone, and the four TestTask cases now run through TaskPool.submit.
S22.3.2 — Measure representative work
State: Closed (2026-11-14)
Measure cold/warm CLI invocation and imports, disabled/enabled logging, sync readiness with slow checks, large Excel/transfer operations, auth key refresh, SQLAlchemy upsert, scaffold setup and strict docs builds. Use deterministic small correctness cases and separately marked larger measurements.
Acceptance:
- A reproducible task-based procedure records environment, input sizes, wall time, peak memory and relevant query/request/build counts, including baseline commit.
- Distinguish CPU, I/O, buffering and tool startup; no external provider or production credentials are required.
- A package records a baseline before optimization and a comparison afterward; noisy wall-time thresholds do not become unit tests.
- Measurements are pytest cases under
packages/<pkg>/tests/measure/, skipped unlessPYKIT_MEASURE_OUTnames an output directory. Each appends one JSON line toresults.jsonlthrough a package-localmeasurefixture. The procedure and per-commit tables live in Performance baselines.
Outcome: baseline recorded for 1a8da87; it is provisional, because other
agent jobs ran on the machine during collection.
S22.3.3 — Reduce repeated test setup where measured
State: Closed (2026-11-16) · Depends on: S22.3.2
Reuse immutable wheel outputs within a run and request only the build closure needed if measurements justify it; keep clean virtual environments for install tests. Keep a real large-input SQL regression even if routine fixtures become smaller.
Acceptance:
- Scaffold cases still install built distributions outside workspace imports and verify relevant extras.
- A build/setup reduction shows fewer operations and an equivalent behavior-case matrix; third-party resolution is not falsely called offline.
- Do not modify generated Taskfiles or add dependencies to obtain a faster test command.
Settled contract: nothing was reduced, because setup is about 1.2 s of an 80 s scaffold run and wheels are already built once per session (10 builds); each install case keeps a fresh venv; third-party packages resolve from the index through the uv cache (not offline); the 33,000-row SQL regression stays.
Outcome: scaffold guards pin one wheel per workspace package, installs
resolving outside packages/, and extras installed only on request (10 cases,
10 builds, 9 installs).
Acceptance
- S22.3.1: Every removed implementation assertion has equivalent public-behavior coverage or a documented reason it was only an architecture check.
- S22.3.2: A reproducible task-based procedure records environment, input sizes, wall time, peak memory and relevant query/request/build counts, including baseline commit.
- S22.3.3: Scaffold cases still install built distributions outside workspace imports and verify relevant extras.