Performance baselines
This guide describes how to collect comparable measurements for representative pykit workloads. Measurements are evidence for later changes; wall time is not a test threshold.
Procedure
Run from the repository root on an otherwise idle machine. Unset VIRTUAL_ENV
before running the commands below. Install the project environment with
task setup. Record the baseline commit, CPU, OS and version, Python version,
and uv version alongside the results. The measurement fixture records the
commit, Python version, platform, inputs, wall time, traced peak memory,
operation counts, and workload kind in JSON Lines. Record CPU and uv version
with the run; the installed uv version can be read from its package installation
metadata without invoking it directly.
Choose an absolute output directory and set PYKIT_MEASURE_OUT for the package
measurements:
This runs all package suites; only measurement cases run when the variable is set. The root scaffold measurement is selected separately:
PYKIT_MEASURE_OUT=/absolute/path/to/results \
SUITE_ARGS='-c pyproject.toml -q packages/rn-forge-cli/tests/measure/test_scaffold.py::test_scaffold_setup' \
task ci:suites
-c pyproject.toml selects the root configuration; the package configuration
deselects scaffold cases. The scaffold case times the session wheel build and
one clean-venv install. The build count comes from the returned wheel set.
Workspace wheels are supplied locally; third-party requirements resolve from the
package index.
Time the strict documentation build separately. On macOS, capture elapsed time and peak resident memory with:
On Linux, use:
The command runs 11 strict builds: the root site and ten package sites. Record the elapsed time and peak resident memory from the platform's time output.
tracemalloc peak memory covers Python allocations in the measured process; it
does not include child processes or native allocations. CLI subprocess cases
record child peak RSS separately. ru_maxrss is reported in bytes on macOS and
KiB on Linux. Keep both values as captured and annotate the units when comparing
platforms.
Workloads
| Package | Measurement | Inputs and counts | Kind |
|---|---|---|---|
| CLI | Cold and warm run(app, ["--help"]), plus python -X importtime -c "import rn_forge.cli" |
Child RSS and import count | Tool startup |
| Commons | Disabled and enabled logging to an in-memory handler | 10,000 records | CPU |
| Web | Sync readiness with eight slow checks | 8 checks at 0.2 seconds each, timeout unset and set | I/O |
| Data | Excel write then load | 50,000 rows × 10 columns; file size | Buffering |
| Data | excel_write: Excel write only, dataset built before measurement |
50,000 rows × 10 columns; bytes, rows, cells | Buffering |
| Tooling | Install a verified local release archive | 64 MiB; download, hash, and extract counts | Buffering |
| Auth | JWT verification against local JWKS server, including one key rotation | 100 verifications; JWKS requests; first, refresh and median cached verification seconds | I/O |
| Auth | Authorization-context cache churn with an injected clock | 1,000 to 16,000 distinct accounts, plus a hot-set case; entries retained, lock hold | CPU |
| Auth | Fill empty in-memory replay, transaction and session stores with live records | 1,000, 4,000 and 16,000 records; 1 hour TTL; operation count | CPU |
| SQLAlchemy | SQLite upsert of initial and mixed rows | 33,000 initial rows and 33,000 mixed rows; SQL statement count | I/O |
| Django | Export and batch get/create/update/delete over the transfer mixins | 100 to 9,000 rows; SQL query count, response bytes | I/O |
| FastAPI | Transfer export, import and batch create driven through the ASGI app | 10,000 and 100,000 rows × 10 columns; response bytes, event-loop lag | I/O |
| CLI scaffold | Built workspace wheels and one clean-venv install | Build count and duration; install duration | Tool startup |
| Docs | Strict root and package builds | 11 builds | Tool startup |
web.transfer contains wire-format helpers and does not call write_xlsx, so
there is no matching transfer export workload. The SQLAlchemy measurement
remains separate from the ordinary test suite's large-input regression.
Baseline record
The initial baseline target is 1a8da87. Measurements were collected on
2026-11-14 at checkout 0a388e917eac36c1cb7ec031b48dd555edefc563, whose changes
from the target commit are confined to spec documents. The package source is
unchanged from 1a8da87.
Environment: Mac mini (Mac14,12), Apple M2 Pro, 10 CPU cores (6 performance and
4 efficiency), 16 GB RAM; macOS 27.2 (26B123), arm64; Python 3.14.2. Installed
uv is 0.13.7, read from the Homebrew installation symlink
(/opt/homebrew/Cellar/uv/0.13.7/bin/uv). The machine was not idle: other agent
jobs ran concurrently. These timings are provisional; the reviewer may re-record
them on an idle machine.
1a8da87 baseline
Wall times are seconds; memory values are bytes. Peak memory is traced Python allocation memory unless labelled child RSS or docs RSS. Counts are observed values, not performance thresholds.
| Workload | Inputs | Wall (s) | Peak (bytes) | Counts |
|---|---|---|---|---|
| Logging enabled | records=10000, enabled=True | 0.157393 | 871,754 | attempted_records=10000, records=10000 |
| Logging disabled | records=10000, enabled=False | 0.005638 | 64 | attempted_records=10000, records=0 |
| CLI warm help | calls=1, warmup_calls=1 | 0.014642 | 111,827 | calls=1, imports=861 |
| CLI import | processes=1 | 0.106406 | 341,302; child RSS 42,418,176 | imports=400 |
| CLI cold help | processes=1 | 0.131324 | 315,297; child RSS 46,809,088 | imports=500 |
| SQLite upsert | initial_rows=33000, mixed_rows=33000 | 1.903825 | 30,528,457 | statements=76 |
| Readiness, no timeout | checks=8, sleep_s=0.2, timeout_s=None | 0.210633 | 64,867 | checks=8, sum_sleeps_s=1.600000 |
| Readiness, 1 s timeout | checks=8, sleep_s=0.2, timeout_s=1.0 | 0.209087 | 56,621 | checks=8, sum_sleeps_s=1.600000 |
| Excel round trip | rows=50000, columns=10 | 22.409027 | 225,496,023 | bytes=1577014, rows=50000, cells=500000 |
| Auth key refresh | verifications=100, rotations=1 | 0.096936 | 290,111 | verifications=100, requests=2 |
| Scaffold setup | requirement=rn-forge-cli | 0.539040 | 710,899 | build_wall_s=0.242943, builds=10, install_wall_s=0.294780, installs=1 |
| Docs | 11 strict builds | 25.39 | docs RSS 146,210,816 | 11 builds |
Docs figures are from /usr/bin/time -l task docs:build: elapsed (real) and
maximum resident set size.
S22.4.4 allocation and algorithm costs
Baseline measurements used detached commit
7f11a7caf0790ac0add12fe782896117456a7b92 with the measurement harness copied
into the checkout. Comparison measurements used
be88ead000e59df8c1a02044863c19999da5821e plus the uncommitted source changes.
Both runs used Python 3.14.2, macOS 27.2.1 (26B131), arm64, and uv 0.13.8 from
its Homebrew installation symlink. Hardware was a Mac mini (Mac14,12), Apple M2
Pro, 10 CPU cores (6 performance and 4 efficiency), 16 GB RAM. The machine's
idle status was not controlled.
Wall times are seconds; peak values are traced Python allocation bytes. Counts
and inputs are identical across each pair; they are observations, not test
thresholds. Raw records: /tmp/S22.4.4-measure-baseline/results.jsonl and
/tmp/S22.4.4-measure-comparison/results.jsonl.
| Workload | Inputs | Baseline wall / peak | Comparison wall / peak | Counts |
|---|---|---|---|---|
| File digest | size_mib=8 | 0.004797 / 8,389,793 | 0.004155 / 395,368 | bytes=8,388,608 |
| File digest | size_mib=64 | 0.033831 / 67,110,053 | 0.032279 / 395,372 | bytes=67,108,864 |
| File digest | size_mib=256 | 0.135940 / 268,436,649 | 0.142800 / 395,376 | bytes=268,435,456 |
| Duplicate check | size=1,000 | 0.026543 / 1,830,392 | 0.013436 / 1,770,312 | names=1,000 |
| Duplicate check | size=4,000 | 0.153058 / 6,831,928 | 0.063503 / 6,917,500 | names=4,000 |
| Duplicate check | size=16,000 | 1.732357 / 27,293,232 | 0.280489 / 27,290,392 | names=16,000 |
| Dict merge, empty target | depth=8, fanout=4 | 0.068877 / 4,683,400 | 0.066504 / 4,683,400 | leaves=65,536 |
| Dict merge, equal target | depth=8, fanout=4 | 0.583678 / 1,670,120 | 0.073448 / 944 | leaves=65,536 |
File digests changed to streaming: traced peak stays near 395 KB for all file sizes instead of growing with the file. Duplicate detection changed to a single-pass count: baseline time grew 5.77× for the first 4× increase in names and 11.32× for the second, with the largest case at 1.732 s; the larger step is below the idealized 16× quadratic ratio because submission overhead is also measured. Deep merge changed to copy on assignment: the equal-target workload improved 87.42% in wall time and 99.94% in peak traced memory, exceeding the 30% rule; the empty-target workload was essentially unchanged. No timing thresholds were added to tests.
7f11a7c before and after S22.12.2
Collected on 2026-11-15 at checkout be88ead000e59df8c1a02044863c19999da5821e.
Before measurement, excel.py was unchanged from 7f11a7c; after measurement,
only the redundant row list was removed. Both runs used the same machine with no
concurrent jobs launched by this agent. Process inspection was denied by the
sandbox, so other machine activity could not be verified.
Environment: Mac mini (Mac14,12), Apple M2 Pro, 10 CPU cores (6 performance and 4 efficiency), 16 GB RAM; macOS 27.2.1 (26B131), arm64; Python 3.14.2; uv 0.13.8.
Each workload uses 50,000 rows and 10 columns. Wall times are seconds; peaks are traced Python allocation bytes. The write-only case builds the dataset before tracing; the unchanged round-trip case includes dataset construction, writing and loading.
| Workload | Phase | Wall (s) | Peak (bytes) | Counts |
|---|---|---|---|---|
excel_write |
Before | 17.990195 | 8,679,433 | bytes=1577015, rows=50000, cells=500000 |
excel_write |
After | 16.409417 | 2,134,784 | bytes=1577015, rows=50000, cells=500000 |
excel_roundtrip |
Before | 24.422749 | 225,584,449 | bytes=1577015, rows=50000, cells=500000 |
excel_roundtrip |
After | 24.269110 | 225,328,754 | bytes=1577014, rows=50000, cells=500000 |
The write-phase traced peak fell by 6,544,649 bytes (75.4%); the round-trip peak fell by 255,695 bytes (0.11%). The caller still holds the complete dataset, and the function returns the complete workbook as bytes.
As with S22.3.2, these are observations, not thresholds or timing assertions.
Single-run timings vary with machine activity; tracemalloc excludes native
allocations and child processes, and these peaks are not process RSS. XLSX ZIP
metadata can vary between saves, including the resulting compressed size; byte
counts do not establish byte-for-byte identity.
S22.9.3 transfer queries and memory
Collected on 2026-11-16 at checkout c85de6ebcbb77d2ac7219413533e27448b867cf9,
with the measurement tests uncommitted and production source unchanged.
Environment: Mac mini (Mac14,12), Apple M2 Pro, 10 CPU cores (6 performance and
4 efficiency), 16 GB RAM; macOS 27.2.1 (26B131), arm64; Python 3.14.2; uv
0.13.8. No concurrent jobs were launched by this agent during the single
measurement run. Process inspection was denied by the sandbox, so machine idle
status could not be verified. Raw records: /tmp/S22.9.3-measure/results.jsonl.
The dialect is SQLite. Wall times are seconds; peaks are traced Python
allocation bytes, not process RSS. Seeding is outside the measurement, and SQL
queries are counted with connection.execute_wrapper, without retaining SQL
strings. These are observations, with no memory or timing thresholds.
| Operation | Rows | wall_s | peak_bytes | queries | response_bytes |
|---|---|---|---|---|---|
| export | 1,000 | 0.057230 | 564,408 | 2 | 15,902 |
| export | 9,000 | 0.443740 | 3,580,505 | 2 | 151,902 |
| batch_get | 100 | 0.007705 | 91,932 | 1 | 2,300 |
| batch_get | 900 | 0.047507 | 708,616 | 1 | 22,300 |
| batch_create | 100 | 0.066071 | 223,712 | 202 | 4,592 |
| batch_create | 900 | 0.512092 | 1,534,374 | 1,802 | 43,901 |
| batch_update | 100 | 0.103678 | 1,177,539 | 103 | 2,110 |
| batch_update | 900 | 0.971890 | 8,608,042 | 903 | 19,710 |
| batch_delete | 100 | 0.026588 | 129,231 | 103 | 0 |
| batch_delete | 900 | 0.210383 | 817,316 | 903 | 0 |
All query counts match the brief's Current behaviour figures. All traced peaks differ from its approximate figures; the largest difference is batch update at 900 rows: 8,608,042 bytes here versus about 6,218,092 bytes in the brief. No source was changed to match those figures.
Writes cost queries per row by design: create 2, update 1, delete 1. Read query
counts do not grow with rows. Export is constant only for a viewset that shapes
its queryset; this measurement uses select_related("team") and related team
rows.
PostgreSQL slope guards also passed in task ci:integration -- rn-forge-django:
824 passed, 5 measurement cases skipped, 1 deselected. Resource measurements
above remain SQLite-only.
d709103 before and after S22.11.2
Measurements were collected on 2026-11-15. The checkout was HEAD
d709103f84c6ba50c0cf8b690ec3fd93991da8a7; after measurements, the S22.11.2
upsert change was uncommitted. Both runs used a Mac mini (Mac14,12), Apple M2
Pro, 10 CPU cores (6 performance and 4 efficiency), 16 GB RAM; macOS 27.2.1
(26B131), arm64; Python 3.14.2; uv 0.13.8; SQLite 3.50.4. Five sequential task
test runs were collected before and after, with no other agent-launched jobs
during measurement. Machine idle status was not verified. Raw records:
/tmp/S22.11.2-before/results.jsonl and /tmp/S22.11.2-after/results.jsonl (20
records each).
Each case used 33,000 rows. Wall times are the median of five runs; peak memory is the maximum observed traced Python allocation peak across five runs. These measurements are evidence, not test thresholds. S22.11.1 set the all-new target at 16 MiB; on 2026-11-16 the owner revised it to at most 27 MiB (28,311,552 bytes), because the remaining peak is the single write list and batching the write would break the statement targets. The mostly unchanged target is at most 12 MiB (12,582,912 bytes), and the statement targets are 38 for single keys and 75 for composite keys.
| Workload | Before median wall (s) | After median wall (s) | Before peak (bytes) | After peak (bytes) | Statements before / after |
|---|---|---|---|---|---|
| Created, single key | 1.317372292 | 1.405893709 | 30,586,048 | 24,642,134 | 38 / 38 |
| Created, composite key | 2.182281083 | 1.929263417 | 31,005,526 | 26,804,635 | 75 / 75 |
| Mixed, single key | 0.636553333 | 0.706001833 | 14,826,115 | 3,835,424 | 38 / 38 |
| Mixed, composite key | 1.018990833 | 1.133083750 | 16,420,648 | 4,163,304 | 75 / 75 |
After the change, both all-new peaks (23.5 and 25.6 MiB) meet the revised target
and both mixed peaks meet 12 MiB; all statement targets were met before and
after. Median wall time changed by +6.72% for created single-key, -11.59% for
created composite-key, +10.91% for mixed single-key, and +11.20% for mixed
composite-key; the owner accepted these in place of the no-median-regression
target. The chunk-only trial peaked at 27,543,664 bytes for single keys and
29,455,034 bytes for composite keys, so audit values were bound once with
insert.values(stamps).
PostgreSQL parity was checked separately with the integration service
(task ci:integration -- rn-forge-sqlalchemy): 167 passed, 2 expected failures
(the documented snapshot race), and only measurement cases skipped.
3027d9d S22.3.3 setup audit — 2026-11-16
Reviewer audit at 3027d9d; environment recorded by the audit: macOS arm64,
Python 3.14, warm uv cache. Other jobs may have shared the machine, so wall
times are noisy. The audit did not record more specific hardware or version
details.
task ci:suites with SUITE_ARGS='-q -m scaffold -c pyproject.toml' ran one
pytest session: 7 scaffold cases passed in 78–85 s, with 2 skipped and 4,533
deselected. The skipped cases were the CLI measurement without
PYKIT_MEASURE_OUT and the Redis test without a Redis URL. The unchanged run
performed 10 wheel builds, 7 venv creations, and 7 installs. The 10 builds took
0.20 s total. The 7 venv creations and installs each took 0.09–0.37 s, 0.97 s
total. Case durations were 59.95 s for the Django conformance driver, 7.49 s for
FastAPI conformance, 2.55 s for FastAPI SSE, 1.50 s for tooling lifecycle, and
0.77, 0.74, and 0.58 s for the three CLI cases. These durations are served
application runtime, not setup.
The unchanged large-input SQL regression writes 33,000 rows, exceeding SQLite's 32,766 parameter limit, in 0.35 s. The new guards leave its input and chunk size unchanged (900).
After adding the three guards, the targeted CLI scaffold run passed 6 cases in 3.40 s. It added 2 venv creations and 2 installs, about 0.8 s; the wheel-set guard installs nothing. The full scaffold run therefore has 10 cases, 10 builds, 9 venv creations, and 9 installs; it passed 10 cases with 2 skipped.
Third-party requirements resolve through the package index using the warm uv cache; workspace wheels are served locally. Build and install setup account for about 1.2 s of about 80 s, and the wall times are noisy. No setup was reduced.
6e0572f before and after S22.8.3
Collected on 2026-11-16 at checkout 6e0572fb99e82033345229fc8f20e111ae115e6a,
before and after the uncommitted S22.8.3 source changes. Environment: Mac mini
(Mac14,12), Apple M2 Pro, 10 CPU cores (6 performance and 4 efficiency), 16 GB
RAM; macOS 27.2.1 (26B131), arm64; Python 3.14.2; uv 0.13.8 from its Homebrew
installation symlink. Five sequential runs per phase used the same machine, with
no other agent-launched jobs during measurement. Machine idle status was not
verified. Raw records: /tmp/S22.8.3-before/results.jsonl and
/tmp/S22.8.3-after/results.jsonl (50 records each).
Wall times are medians of five runs; peaks are the maximum traced Python allocation bytes. Each store starts empty and receives distinct live records with a one-hour TTL. Counts are observations, not test thresholds.
| Workload | Records | Before wall (s) | After wall (s) | Before peak (bytes) | After peak (bytes) | Operations |
|---|---|---|---|---|---|---|
| Replay | 1,000 | 0.021692666 | 0.002485500 | 131,530 | 92,582 | 1,000 |
| Replay | 4,000 | 0.286680333 | 0.009467375 | 533,314 | 377,406 | 4,000 |
| Replay | 16,000 | 4.587994875 | 0.038046667 | 2,144,914 | 1,524,358 | 16,000 |
| Transaction | 1,000 | 0.027857125 | 0.007170250 | 403,266 | 304,693 | 1,000 |
| Transaction | 4,000 | 0.342122208 | 0.037973042 | 1,622,402 | 1,467,197 | 4,000 |
| Transaction | 16,000 | 5.207494334 | 0.142185834 | 6,428,874 | 5,748,549 | 16,000 |
| Session | 1,000 | 0.032801917 | 0.007477584 | 430,620 | 391,820 | 1,000 |
| Session | 4,000 | 0.404766500 | 0.035189916 | 1,589,308 | 1,433,772 | 4,000 |
| Session | 16,000 | 6.104716000 | 0.155544375 | 6,797,244 | 6,238,828 | 16,000 |
Key refresh performs 100 verifications and one rotation. Single-call latencies below are medians across five runs; cached latency is the median across runs of each run's median of the other 98 verifications. Both phases made exactly two JWKS requests in every run.
| Phase | Wall (s) | Peak (bytes) | First verify (s) | Refresh verify (s) | Cached verify median (s) | Requests |
|---|---|---|---|---|---|---|
| Before | 0.101847458 | 302,546 | 0.010706542 | 0.005855458 | 0.000133480 | 2 |
| After | 0.092315791 | 300,106 | 0.010232708 | 0.005510708 | 0.000129167 | 2 |
S22.8.4 authorization-context churn baseline
Five sequential runs were collected on 2026-11-16 with production source
unchanged, including context_cache.py (also unchanged from the brief's
01a8985). The measurement tests were uncommitted. No concurrent jobs were
launched by this agent during measurement; machine idle status was not verified.
| Checkout commit | CPU and hardware | OS | Python | uv |
|---|---|---|---|---|
d0a1a8085dd01d7db871d9a7413b4ad91269f0fd |
Mac mini (Mac14,12), Apple M2 Pro, 10 cores (6 performance, 4 efficiency), 16 GB RAM | macOS 27.2.1 (26B131), arm64 | 3.14.2 | 0.13.8 (Homebrew installation symlink) |
Raw records: /tmp/S22.8.4-baseline-five/run-{1,2,3,4,5}/results.jsonl, five
records per run. Earlier exploratory records under /tmp/S22.8.4-baseline are
excluded. Each step advances the injected clock by one second and resolves a
fresh account, with a fresh context on each provider call and a 60 s TTL. The
invalidated case also invalidates that account after resolving it. The hot case
additionally resolves one of 20 recurring accounts per step. A hit does not
refresh stored_at.
Wall times and counts below are medians across five runs; peak memory is the
maximum across runs. Memory is traced Python allocation bytes, not process RSS.
Lock holds run under tracemalloc, so their absolute durations are inflated and
comparable only within this harness. Only the cache's entry lock is timed, not
its version-counter lock. These figures are observations, not thresholds.
| Case | Accounts / steps | Wall (s) | Peak bytes (max) | Retained bytes (median) |
|---|---|---|---|---|
| Plain | 1,000 | 0.008551709 | 437,212 | 436,878 |
| Plain | 4,000 | 0.030114625 | 1,732,990 | 1,598,550 |
| Plain | 16,000 | 0.137117000 | 6,846,758 | 6,787,846 |
| Invalidated | 16,000 | 0.140850084 | 7,261,894 | 7,202,870 |
| Hot (20 accounts) | 16,000 | 0.219739542 | 6,927,852 | 6,797,548 |
| Case | Accounts / steps | Entries q1 / q2 / q3 / q4 | Entries end | TTL-live end | Version accounts | Provider calls |
|---|---|---|---|---|---|---|
| Plain | 1,000 | 250 / 500 / 750 / 1,000 | 1,000 | 60 | — | — |
| Plain | 4,000 | 1,000 / 2,000 / 3,000 / 4,000 | 4,000 | 60 | — | — |
| Plain | 16,000 | 4,000 / 8,000 / 12,000 / 16,000 | 16,000 | 60 | — | — |
| Invalidated | 16,000 | 4,000 / 8,000 / 12,000 / 16,000 | 16,000 | 60 | 16,000 | — |
| Hot (20 accounts) | 16,000 | 4,020 / 8,020 / 12,020 / 16,020 | 16,020 | 80 | — | 21,340 |
| Case | Accounts / steps | Lock acquisitions | Hold total (s, median) | Hold max (s, median) | Hold max (s, maximum) |
|---|---|---|---|---|---|
| Plain | 1,000 | 2,000 | 0.001504556 | 0.000009916 | 0.000020708 |
| Plain | 4,000 | 8,000 | 0.004812193 | 0.000014916 | 0.000023375 |
| Plain | 16,000 | 32,000 | 0.021412113 | 0.000050791 | 0.000076292 |
| Invalidated | 16,000 | 32,000 | 0.022043150 | 0.000044458 | 0.000055959 |
| Hot (20 accounts) | 16,000 | 53,340 | 0.034928622 | 0.000049667 | 0.000076541 |
Entries grow with distinct accounts ever requested, despite only 60 plain entries remaining younger than the TTL. The hot case makes 16,000 cold calls and 5,340 hot calls: a hot account recurs every 20 s but still expires every 60 s. Invalidation leaves 16,000 entries and 16,000 version counters; its TTL-live count measures age only, not version validity. Its memory figures include both entries and counters and do not isolate counter allocation.
S22.8.1 authorization-context churn after the change
Five sequential runs were collected on 2026-11-16 at checkout
25cbb473093df3d6f0ec4a4d5435d20424971689, with the S22.8.1 changes
uncommitted. Environment: Mac mini (Mac14,12), Apple M2 Pro, 10 CPU cores (6
performance and 4 efficiency), 16 GB RAM; macOS 27.2.1 (26B131), arm64; Python
3.14.2; uv 0.13.8 (reported by the validation build). No other jobs were
launched by this agent during measurement; machine idle status was not verified.
Each run used
unset VIRTUAL_ENV; PYKIT_MEASURE_OUT=/tmp/S22.8.1-after/run-$n PYTEST_ADDOPTS='tests/measure/test_context_churn.py' task test PACKAGE_DIRS=packages/rn-forge-auth
for n=1,2,3,4,5. Each exited 0 with five passed cases and five records. Raw
records: /tmp/S22.8.1-after/run-{1,2,3,4,5}/results.jsonl (25 total).
The workload and harness are unchanged from S22.8.4: a 60 s TTL, one second per
fresh account, with invalidation or 20 recurring accounts where indicated.
Paired cells below show S22.8.4 baseline → S22.8.1 after; single values are
unchanged. Wall times and counts are medians of five runs; peak bytes and the
last lock column are maxima across runs. Memory is traced Python allocation, not
process RSS. Lock holds are inflated by tracemalloc and compare only within
this harness; only the entry lock is timed, not the version-counter lock. These
observations are not thresholds.
| Case | Accounts / steps | Wall (s) | Peak bytes (max) | Retained bytes (median) |
|---|---|---|---|---|
| Plain | 1,000 | 0.008551709 → 0.008212417 | 437,212 → 33,877 | 436,878 → 25,152 |
| Plain | 4,000 | 0.030114625 → 0.033203375 | 1,732,990 → 33,937 | 1,598,550 → 25,212 |
| Plain | 16,000 | 0.137117000 → 0.127250125 | 6,846,758 → 28,277 | 6,787,846 → 25,272 |
| Invalidated | 16,000 | 0.140850084 → 0.137095542 | 7,261,894 → 1,185,100 | 7,202,870 → 1,178,366 |
| Hot (20 accounts) | 16,000 | 0.219739542 → 0.205115709 | 6,927,852 → 41,459 | 6,797,548 → 31,518 |
| Case | Accounts / steps | Entries q1 / q2 / q3 / q4 | Entries end | TTL-live end | Version accounts | Provider calls |
|---|---|---|---|---|---|---|
| Plain | 1,000 | 250 / 500 / 750 / 1,000 → 60 / 60 / 60 / 60 | 1,000 → 60 | 60 | — | — |
| Plain | 4,000 | 1,000 / 2,000 / 3,000 / 4,000 → 60 / 60 / 60 / 60 | 4,000 → 60 | 60 | — | — |
| Plain | 16,000 | 4,000 / 8,000 / 12,000 / 16,000 → 60 / 60 / 60 / 60 | 16,000 → 60 | 60 | — | — |
| Invalidated | 16,000 | 4,000 / 8,000 / 12,000 / 16,000 → 60 / 60 / 60 / 60 | 16,000 → 60 | 60 | 16,000 | — |
| Hot (20 accounts) | 16,000 | 4,020 / 8,020 / 12,020 / 16,020 → 80 / 80 / 80 / 80 | 16,020 → 80 | 80 | — | 21,340 |
| Case | Accounts / steps | Lock acquisitions | Hold total (s, median) | Hold max (s, median) | Hold max (s, maximum) |
|---|---|---|---|---|---|
| Plain | 1,000 | 2,000 | 0.001504556 → 0.002909271 | 0.000009916 → 0.000016958 | 0.000020708 → 0.000023250 |
| Plain | 4,000 | 8,000 | 0.004812193 → 0.012166388 | 0.000014916 → 0.000027333 | 0.000023375 → 0.000043417 |
| Plain | 16,000 | 32,000 | 0.021412113 → 0.046389403 | 0.000050791 → 0.000041458 | 0.000076292 → 0.000077416 |
| Invalidated | 16,000 | 32,000 | 0.022043150 → 0.046045244 | 0.000044458 → 0.000043042 | 0.000055959 → 0.000648375 |
| Hot (20 accounts) | 16,000 | 53,340 | 0.034928622 → 0.066478913 | 0.000049667 → 0.000056542 | 0.000076541 → 0.000070542 |
Median lock hold totals are higher in every case: plain 1,000 by 0.001404715 s (93.4%), plain 4,000 by 0.007354195 s (152.8%), plain 16,000 by 0.024977290 s (116.7%), invalidated by 0.024002094 s (108.9%), and hot by 0.031550291 s (90.3%). The sweep adds work under the existing lock. Median maximum holds are higher for plain 1,000 by 0.000007042 s (71.0%), plain 4,000 by 0.000012417 s (83.2%), and hot by 0.000006875 s (13.8%); they are lower for plain 16,000 by 0.000009333 s (18.4%) and invalidated by 0.000001416 s (3.2%). Acquisitions are unchanged in all five cases.
After a miss, entries are bounded by the accounts resolved within one TTL: plain
churn retains 60 entries, all TTL-live; hot churn retains 80, all TTL-live. An
idle cache keeps its last entries until the next miss, and hits do not refresh
stored_at. A burst of N distinct accounts inside one TTL retains N entries;
there is no hard cap. Concurrent inserts can leave an expired entry behind a
live front entry until a later sweep reaches it.
Version counters are not pruned: invalidated churn still retains 16,000 counters and 1,178,366 B (about 1.18 MB), despite only 60 entries. TTL-live counts measure age, not version validity; retained memory includes both entries and counters and does not isolate counter allocation.
Single-flight was not added: there is no measured need. Hot churn still makes 21,340 provider calls (16,000 cold and 5,340 hot), and the sweep adds neither provider calls nor lock acquisitions. Concurrent misses for the same account still each call the provider.
9bcf9b0 S22.10.3 FastAPI transfer baseline
Collected on 2026-11-16 at commit 9bcf9b0e6b4b6bb4a4c7ad2ec409e30a7f62d17a,
with the measurement tests uncommitted and production source unchanged.
Environment: Mac mini (Mac14,12), Apple M2 Pro, 10 cores (6 performance and 4
efficiency), 16 GB RAM; macOS 27.2.1 (26B131), arm64; Python 3.14.2; uv 0.13.8.
No other jobs were launched by this agent during the three sequential runs.
Machine-wide idle status could not be verified: process inspection was denied by
the sandbox.
The harness runs under tracemalloc, which inflates the XLSX cases: S25.2.1
found 100,000-row XLSX export about 5× and import about 2.5× slower than
untraced (about 6.5 s and 8 s untraced, on 2026-11-18).
Exact first command; repeated with output directories /tmp/S22.10.3-run-2 and
/tmp/S22.10.3-run-3:
unset VIRTUAL_ENV
PYKIT_MEASURE_OUT=/tmp/S22.10.3-run-1 \
PYTEST_ADDOPTS='tests/measure/test_contract.py tests/measure/test_transfer.py -p no:randomly' \
task test PACKAGE_DIRS=packages/rn-forge-fastapi
Raw records: /tmp/S22.10.3-run-{1,2,3}/results.jsonl (11 each).
| Case | Format / rows / dry-run | Wall median / max (s) | Peak max (bytes) | Lag max median / max (s) | p99 median (s) | Heartbeats range | Response bytes range |
|---|---|---|---|---|---|---|---|
fastapi_export |
csv / 10,000 / — | 1.249249 / 1.255332 | 235,123 | 0.010022 / 0.011498 | 0.000131 | 246–248 | 687,837 |
fastapi_export |
csv / 100,000 / — | 12.141380 / 12.163089 | 264,607 | 0.009431 / 0.009930 | 0.000099 | 2,396–2,402 | 7,077,837 |
fastapi_export |
xlsx / 10,000 / — | 3.462552 / 3.491984 | 3,471,279 | 0.016410 / 0.016427 | 0.016373 | 187–190 | 378,148–378,149 |
fastapi_export |
xlsx / 100,000 / — | 30.915327 / 31.064210 | 29,350,992 | 0.021964 / 0.037788 | 0.016390 | 1,625–1,632 | 3,742,582–3,742,583 |
fastapi_import |
csv / 10,000 / false | 0.147006 / 0.147787 | 23,681,025 | 0.018863 / 0.020379 | 0.016368† | 8† | 62 |
fastapi_import |
csv / 100,000 / false | 1.466050 / 1.529704 | 229,128,617 | 0.131530 / 0.131557 | 0.047659† | 63–67† | 63 |
fastapi_import |
xlsx / 10,000 / false | 1.952598 / 1.963844 | 20,929,404 | 0.107315 / 0.123496 | 0.090906† | 42–56† | 62 |
fastapi_import |
xlsx / 100,000 / false | 19.168049 / 19.180899 | 204,092,667 | 0.300066 / 0.300617 | 0.136018 | 406–411 | 63 |
fastapi_import |
csv / 100,000 / true | 1.527289 / 1.535227 | 229,131,627 | 0.118767 / 0.120848 | 0.072601† | 65† | 62 |
fastapi_batch_create |
json / 10,000 / — | 0.263132 / 0.267298 | 35,500,680 | 0.151127 / 0.152035 | 0.121079† | 2† | 1,577,790 |
fastapi_batch_create |
json / 100,000 / — | 2.603145 / 2.612255 | 354,722,482 | 1.337377 / 1.338772 | 1.272576† | 2† | 15,977,790 |
All cases have columns=10 and kind="io"; all 33 request statuses were 200.
Wall times and p99 are medians of three runs; peaks and the second maximum lag
figure are maxima across runs. Heartbeats and response sizes show observed
ranges. Tracing starts after building the app, source rows and request body; the
ASGI send callback counts and discards response chunks without buffering.
† Fewer than 100 heartbeats: p99 is not meaningful; use the maximum lag. The heartbeat sleeps 5 ms on the app's own loop, settles for three intervals, and records oversleep until completion. Worker-thread GIL contention remains visible, including the roughly 16 ms p99 floor on XLSX export.
Request sizes (inputs.request_bytes, including multipart framing) were: CSV
import 687,973 / 7,077,973 bytes for 10,000 / 100,000 rows, with the same
7,077,973 for dry-run; XLSX import 378,582–378,586 / 3,742,998 bytes; batch
create 1,577,794 / 15,977,794 bytes. XLSX ZIP metadata can change file sizes
between runs. Input rows are deterministic: sequential ID and SKU, Widget,
Standard item, North, Ops, quantity 3, price 12.5, due 2026-01-02, active true.
cd0b069 S22.10.2 FastAPI transfer after the change
Collected on 2026-11-16 on commit cd0b06958c72c0b4dd20ed200e9d8085b297429d
with the S22.10.2 change uncommitted. Environment: Mac mini (Mac14,12), Apple M2
Pro, 10 cores, 16 GB RAM; macOS 27.2.1, arm64; Python 3.14.2; uv 0.13.8. No
other jobs were launched by this agent during the three sequential runs.
Machine-wide idle status could not be verified.
Exact first command; repeated with output directories /tmp/S22.10.2-run-2 and
/tmp/S22.10.2-run-3:
unset VIRTUAL_ENV
PYKIT_MEASURE_OUT=/tmp/S22.10.2-run-1 \
PYTEST_ADDOPTS='tests/measure/test_contract.py tests/measure/test_transfer.py -p no:randomly' \
task test PACKAGE_DIRS=packages/rn-forge-fastapi
Raw records: /tmp/S22.10.2-run-{1,2,3}/results.jsonl (11 each). Each run was
12 passed; all 33 request statuses were 200. Response sizes are unchanged from
the S22.10.3 baseline: 7,077,837 bytes for CSV export and 15,977,790 for batch
create at 100,000 rows. Before and After are medians of three runs; Peak after
is the maximum. Bounds are the metrics judged: wall for CSV export, peak for CSV
export, CSV import and dry-run and batch create, and lag for batch create; every
other case is judged at 1.25 x its baseline median wall and lag (lag never below
0.016 s).
| Case | Format / rows / dry-run | Bound | Before | After | Peak after (bytes) | Met |
|---|---|---|---|---|---|---|
fastapi_export |
csv / 10,000 / — | wall at most 0.30 s; peak at most 1,048,576 | wall 1.249249 s; peak 235,123 | wall 0.088818 s | 646,969 | yes |
fastapi_export |
csv / 100,000 / — | wall at most 3.0 s; peak at most 1,048,576 | wall 12.141380 s; peak 264,607 | wall 0.793433 s | 626,730 | yes |
fastapi_export |
xlsx / 10,000 / — | no bound set; wall at most 4.328 s, lag at most 0.0205 s | wall 3.462552 s; lag 0.016410 s | wall 3.442899 s; lag 0.016441 s | 3,470,227 | yes |
fastapi_export |
xlsx / 100,000 / — | no bound set; wall at most 38.644 s, lag at most 0.0275 s | wall 30.915327 s; lag 0.021964 s | wall 31.331633 s; lag 0.020656 s | 29,352,061 | yes |
fastapi_import |
csv / 10,000 / false | peak at most 26,214,400; wall at most 0.184 s, lag at most 0.0236 s | wall 0.147006 s; lag 0.018863 s; peak 23,681,025 | wall 0.140319 s; lag 0.022520 s | 23,669,863 | yes |
fastapi_import |
csv / 100,000 / false | peak at most 241,172,480; wall at most 1.833 s, lag at most 0.1644 s | wall 1.466050 s; lag 0.131530 s; peak 229,128,617 | wall 1.480388 s; lag 0.140263 s | 229,128,185 | yes |
fastapi_import |
xlsx / 10,000 / false | no bound set; wall at most 2.441 s, lag at most 0.1341 s | wall 1.952598 s; lag 0.107315 s | wall 1.894994 s; lag 0.123968 s | 20,928,497 | yes |
fastapi_import |
xlsx / 100,000 / false | no bound set; wall at most 23.960 s, lag at most 0.3751 s | wall 19.168049 s; lag 0.300066 s | wall 18.165460 s; lag 0.291586 s | 204,089,659 | yes |
fastapi_import |
csv / 100,000 / true | peak at most 241,172,480; wall at most 1.909 s, lag at most 0.1485 s | wall 1.527289 s; lag 0.118767 s; peak 229,131,627 | wall 1.439095 s; lag 0.109699 s | 229,131,220 | yes |
fastapi_batch_create |
json / 10,000 / — | peak at most 37,748,736; lag at most 0.075 s | lag 0.151127 s; peak 35,500,680 | lag 0.044876 s; wall 0.095824 s | 22,492,295 | yes |
fastapi_batch_create |
json / 100,000 / — | peak at most 367,001,600; lag at most 0.750 s | lag 1.337377 s; peak 354,722,482 | lag 0.504502 s; wall 0.972012 s | 224,796,916 | yes |
Every case met its bound. The batch create lag of 0.5045 s at 100,000 rows is the thread parse of the 15,977,794-byte body. The CSV import lag at 10,000 rows (0.0225 s against 0.0236 s) is the closest to a bound among the cases judged at 1.25 x baseline.
b1949212 S25.2.2 faster XLSX path
Collected on 2026-11-18 at commit b1949212, with the S25.2.2 change
uncommitted. Environment: Mac mini (Mac14,12), Apple M2 Pro, 10 cores, 16 GB
RAM; macOS 27.2.1, arm64; Python 3.14.2. Other processes were running (an editor
and a macOS background scan were among the top CPU users), and machine-wide idle
status could not be verified.
Method, untraced: one temporary task test runner built 100,000 rows × 10
columns of the S22.10.3 row shape, then in one process ran, back to back and
three times each, the unchanged write_xlsx (copied from b1949212) and the
new one, each followed by tablib.Dataset().load(bytes, format="xlsx") of its
output. The runner was deleted afterwards. Harness cases ran as in the procedure
above, with PYKIT_MEASURE_OUT set, once before and once after the change,
under tracemalloc.
| Operation (100,000 × 10, untraced) | Before median (range) s | After median (range) s | Output bytes |
|---|---|---|---|
write_xlsx |
6.019 (6.013–6.030) | 4.398 (4.379–4.423) | 3,742,582 / 3,742,603 |
tablib xlsx load |
7.055 (7.034–7.075) | 4.752 (4.746–4.814) | — |
| Harness case | Wall before / after (s) | Peak before / after (bytes) |
|---|---|---|
excel_write (50,000 rows) |
15.258 / 11.493 | 2,134,607 / 2,133,747 |
excel_roundtrip (50,000 rows) |
23.119 / 18.803 | 225,329,169 / 225,329,638 |
fastapi_export xlsx 10,000 |
3.211 / 2.933 | 3,504,436 / 3,522,166 |
fastapi_export xlsx 100,000 |
31.534 / 25.253 | 29,354,558 / 29,370,898 |
fastapi_import xlsx 10,000 |
1.831 / 1.783 | 20,995,131 / 20,996,764 |
fastapi_import xlsx 100,000 |
18.532 / 18.407 | 204,120,189 / 204,120,455 |
Harness wall times are single runs under tracemalloc and are inflated. Peak
allocation did not fall in four cases; the largest rise is 17,730 bytes (0.5%)
on the 10,000-row export. The fastapi_import cases build their upload with
tablib's own XLSX export, not write_xlsx, so they carry no <dimension> and
show no import gain; only files write_xlsx writes load faster.