F25.2 — Faster XLSX transfer
| State | Closed |
| Parent | E25 |
| Iteration | Release 10 |
| Start Date | 2026-11-18 |
| Closed Date | 2026-11-18 |
| Predecessors | — |
| Source | S22.10.3, owner 2026-11-16 |
Stories: 2 Closed
| Story | State | Depends on |
|---|---|---|
| S25.2.1 — Profile XLSX transfer and choose an approach | Closed (2026-11-18) | — |
| S25.2.2 — Implement the faster XLSX path | Closed (2026-11-18) | S25.2.1 |
rn-forge-data writes XLSX faster for large tables, and the files it writes
load faster, with no change to what a caller gets back.
Context
S22.10.3 measured FastAPI XLSX export at about 31 s and import at about 19 s for
100,000 rows × 10 columns, nearly all of it in rn_forge.data.write_xlsx and
tablib's XLSX load. S25.2.1 found that the harness's tracemalloc inflates
these about 5× and 2.5×: untraced, they are about 6.5 s and 8 s. S22.10.3's
figures are in the
S22.10.3 baseline.
The cost sits in the data package, outside route assembly, so
F22.10 left it out of scope.
S25.2.1 — Profile XLSX transfer and choose an approach
State: Closed (2026-11-18)
Acceptance:
- The story records where the time goes for
write_xlsxand for the XLSX load, from a profile at 100,000 × 10, with the profile's commit. - It compares the candidate approaches, including a write-only or streaming workbook and a different reader or writer, by measured time at 100,000 × 10 and by memory.
- It names the chosen approach and a target time at 100,000 × 10 for export and for import, taken from those measurements.
- It states whether any public API changes; when one does, it names the symbols.
Settled contract (owner, 2026-11-18):
- Where the time goes (profiled at
5e144716): export creates aWriteOnlyCellfor every value, andet_xmlfileserialises each one. A write-only sheet carries no<dimension>, so openpyxl's read-only loader parses the whole sheet twice on import; the first pass is about a third of the import. - Approach: in
write_xlsxonly. Unformatted values are appended raw, andWriteOnlyCellis created only for columns named incolumn_formats. The sheet declares its dimension (A1:<last column><rows + header>) before rows are appended, through the write-only sheet'scalculate_dimensionhook, which openpyxl 3.1.5 reads but does not document; a test guards it. - Unchanged:
write_xlsx's signature, header handling,KeyErrorfor unknowncolumn_formatsnames, and the values read back. tablib still loads imports;rn-forge-fastapidoes not change. No public API changes and no new dependency. - Target at 100,000 × 10, untraced, on the S22.10.3 machine, compared back-to-back with the unchanged code in one session: export at most 4.8 s, import at most 5.5 s.
- Rejected: reading without tablib (about 4% faster, and short rows read as
Noneinstead of''); lower zip compression (0.1–0.2 s for a 27% larger file, through openpyxl internals);lxml,xlsxwriterandpython-calamine(new dependencies, not measured).
Outcome: untraced at 5e144716, export took 6.1–6.7 s and import 7.9 s; the
prototype took 4.6 s and 5.3 s. Memory was not compared per option; S25.2.2's
before and after harness runs record it.
S25.2.2 — Implement the faster XLSX path
State: Closed (2026-11-18) · Depends on: S25.2.1
Acceptance:
- The existing S22.10.3 XLSX export and import workloads are re-measured before the change and after it, on the same machine and by the same procedure; both sets of figures are recorded in the performance baselines.
- The after figures meet the target S25.2.1 set, untraced and compared back-to-back with the unchanged code; the harness's peak allocation for each case rises by no more than 1%.
- A test asserts that the written sheet declares its
<dimension>, so an openpyxl upgrade that drops the hook fails the gate. The gate has no timing assertion. - Behaviour is preserved: the existing tests pass unchanged, and a test pins
that a round trip returns the same values and number formats, with and
without
column_formats, and for an empty dataset and one without headers. - The package's
CHANGELOG.mdhas an entry under[Unreleased]. task validatepasses.
Outcome: untraced at 100,000 × 10, write_xlsx took 4.40 s (from 6.02 s)
and tablib's load of its output 4.75 s (from 7.06 s); the largest peak
allocation rise was 0.5%, on the 10,000-row FastAPI export.
Acceptance
The before and after figures in the baselines guide are read, not scripted.