Skip to content

F22.3 — Behavioral tests and performance baselines

State Closed
Parent E22
Iteration Release 9
Start Date 2026-11-14
Closed Date 2026-11-16
Predecessors —

Stories: 3 Closed

Story State Depends on
S22.3.1 — Keep contract tests independent of source layout Closed (2026-11-14) —
S22.3.2 — Measure representative work Closed (2026-11-14) —
S22.3.3 — Reduce repeated test setup where measured Closed (2026-11-16) S22.3.2

Protect the behavior being simplified and measure expensive work before changing it.

Context

The suite is mostly named by behavior already: only nine test names carry story-like IDs, all in commons block tests. Root built_wheels builds all ten packages once per scaffold session; scaffold environments resolve third-party requirements from the index. SQLAlchemy upsert tests write 33,000 rows. Commons task tests directly call _start, renaming the pytest main thread, and inject an impossible orphan into private pool state. These are concrete test-coupling cases; setup/large-input work remains a measurement target.

S22.3.1 — Keep contract tests independent of source layout

State: Closed (2026-11-14)

Inventory private patches, source-inspection assertions, duplicate test builders and time-sensitive assertions only in areas this epic changes. Rename the nine block tests by behavior. Preserve architecture checks that intentionally inspect imports or examples.

Acceptance:

  • Every removed implementation assertion has equivalent public-behavior coverage or a documented reason it was only an architecture check.
  • Collected behavioral cases are preserved, including parameter variants, negative/security cases and optional-extra installs.
  • Task execution tests use submit/futures/shutdown and preserve the test runner thread name; timing tests use synchronization or controllable clocks where possible, without blanket deletion of sleeps or AST checks.
  • The orphan-future shutdown test is deleted; TaskPool.shutdown's future is None branch stays for F22.4. The map_settled cancellation test keeps its private queue poll, because no public API exposes queued state.

Outcome: commons collects 1,057 cases, down from 1,058: only the orphan test is gone, and the four TestTask cases now run through TaskPool.submit.

S22.3.2 — Measure representative work

State: Closed (2026-11-14)

Measure cold/warm CLI invocation and imports, disabled/enabled logging, sync readiness with slow checks, large Excel/transfer operations, auth key refresh, SQLAlchemy upsert, scaffold setup and strict docs builds. Use deterministic small correctness cases and separately marked larger measurements.

Acceptance:

  • A reproducible task-based procedure records environment, input sizes, wall time, peak memory and relevant query/request/build counts, including baseline commit.
  • Distinguish CPU, I/O, buffering and tool startup; no external provider or production credentials are required.
  • A package records a baseline before optimization and a comparison afterward; noisy wall-time thresholds do not become unit tests.
  • Measurements are pytest cases under packages/<pkg>/tests/measure/, skipped unless PYKIT_MEASURE_OUT names an output directory. Each appends one JSON line to results.jsonl through a package-local measure fixture. The procedure and per-commit tables live in Performance baselines.

Outcome: baseline recorded for 1a8da87; it is provisional, because other agent jobs ran on the machine during collection.

S22.3.3 — Reduce repeated test setup where measured

State: Closed (2026-11-16) · Depends on: S22.3.2

Reuse immutable wheel outputs within a run and request only the build closure needed if measurements justify it; keep clean virtual environments for install tests. Keep a real large-input SQL regression even if routine fixtures become smaller.

Acceptance:

  • Scaffold cases still install built distributions outside workspace imports and verify relevant extras.
  • A build/setup reduction shows fewer operations and an equivalent behavior-case matrix; third-party resolution is not falsely called offline.
  • Do not modify generated Taskfiles or add dependencies to obtain a faster test command.

Settled contract: nothing was reduced, because setup is about 1.2 s of an 80 s scaffold run and wheels are already built once per session (10 builds); each install case keeps a fresh venv; third-party packages resolve from the index through the uv cache (not offline); the 33,000-row SQL regression stays.

Outcome: scaffold guards pin one wheel per workspace package, installs resolving outside packages/, and extras installed only on request (10 cases, 10 builds, 9 installs).

Acceptance

  • S22.3.1: Every removed implementation assertion has equivalent public-behavior coverage or a documented reason it was only an architecture check.
  • S22.3.2: A reproducible task-based procedure records environment, input sizes, wall time, peak memory and relevant query/request/build counts, including baseline commit.
  • S22.3.3: Scaffold cases still install built distributions outside workspace imports and verify relevant extras.