Skip to content

F25.2 — Faster XLSX transfer

State Closed
Parent E25
Iteration Release 10
Start Date 2026-11-18
Closed Date 2026-11-18
Predecessors —
Source S22.10.3, owner 2026-11-16

Stories: 2 Closed

Story State Depends on
S25.2.1 — Profile XLSX transfer and choose an approach Closed (2026-11-18) —
S25.2.2 — Implement the faster XLSX path Closed (2026-11-18) S25.2.1

rn-forge-data writes XLSX faster for large tables, and the files it writes load faster, with no change to what a caller gets back.

Context

S22.10.3 measured FastAPI XLSX export at about 31 s and import at about 19 s for 100,000 rows × 10 columns, nearly all of it in rn_forge.data.write_xlsx and tablib's XLSX load. S25.2.1 found that the harness's tracemalloc inflates these about 5× and 2.5×: untraced, they are about 6.5 s and 8 s. S22.10.3's figures are in the S22.10.3 baseline. The cost sits in the data package, outside route assembly, so F22.10 left it out of scope.

S25.2.1 — Profile XLSX transfer and choose an approach

State: Closed (2026-11-18)

Acceptance:

  • The story records where the time goes for write_xlsx and for the XLSX load, from a profile at 100,000 × 10, with the profile's commit.
  • It compares the candidate approaches, including a write-only or streaming workbook and a different reader or writer, by measured time at 100,000 × 10 and by memory.
  • It names the chosen approach and a target time at 100,000 × 10 for export and for import, taken from those measurements.
  • It states whether any public API changes; when one does, it names the symbols.

Settled contract (owner, 2026-11-18):

  • Where the time goes (profiled at 5e144716): export creates a WriteOnlyCell for every value, and et_xmlfile serialises each one. A write-only sheet carries no <dimension>, so openpyxl's read-only loader parses the whole sheet twice on import; the first pass is about a third of the import.
  • Approach: in write_xlsx only. Unformatted values are appended raw, and WriteOnlyCell is created only for columns named in column_formats. The sheet declares its dimension (A1:<last column><rows + header>) before rows are appended, through the write-only sheet's calculate_dimension hook, which openpyxl 3.1.5 reads but does not document; a test guards it.
  • Unchanged: write_xlsx's signature, header handling, KeyError for unknown column_formats names, and the values read back. tablib still loads imports; rn-forge-fastapi does not change. No public API changes and no new dependency.
  • Target at 100,000 × 10, untraced, on the S22.10.3 machine, compared back-to-back with the unchanged code in one session: export at most 4.8 s, import at most 5.5 s.
  • Rejected: reading without tablib (about 4% faster, and short rows read as None instead of ''); lower zip compression (0.1–0.2 s for a 27% larger file, through openpyxl internals); lxml, xlsxwriter and python-calamine (new dependencies, not measured).

Outcome: untraced at 5e144716, export took 6.1–6.7 s and import 7.9 s; the prototype took 4.6 s and 5.3 s. Memory was not compared per option; S25.2.2's before and after harness runs record it.

S25.2.2 — Implement the faster XLSX path

State: Closed (2026-11-18) · Depends on: S25.2.1

Acceptance:

  • The existing S22.10.3 XLSX export and import workloads are re-measured before the change and after it, on the same machine and by the same procedure; both sets of figures are recorded in the performance baselines.
  • The after figures meet the target S25.2.1 set, untraced and compared back-to-back with the unchanged code; the harness's peak allocation for each case rises by no more than 1%.
  • A test asserts that the written sheet declares its <dimension>, so an openpyxl upgrade that drops the hook fails the gate. The gate has no timing assertion.
  • Behaviour is preserved: the existing tests pass unchanged, and a test pins that a round trip returns the same values and number formats, with and without column_formats, and for an empty dataset and one without headers.
  • The package's CHANGELOG.md has an entry under [Unreleased].
  • task validate passes.

Outcome: untraced at 100,000 × 10, write_xlsx took 4.40 s (from 6.02 s) and tablib's load of its output 4.75 s (from 7.06 s); the largest peak allocation rise was 0.5%, on the 10,000-row FastAPI export.

Acceptance

set -euo pipefail
task validate   # S25.2.1, S25.2.2

The before and after figures in the baselines guide are read, not scripted.