Skip to content

F17.6 — Durable in-process job queue

State New
Parent E17
Tags deferred
Predecessors F17.4, F6.2
Source Owner request, 2026-10-21
Entry criteria a second application needs the queue

Stories: 3 New

Story State Depends on
S17.6.1 — Settle the shared surface with a second application New —
S17.6.2 — Build the models, worker and context New —
S17.6.3 — Build the SSE delta stream New —

A single-user application that runs long work (a library scan, a bulk edit, a download) needs that work to survive a restart, report progress, be cancelled part-way and show its log, without a broker to install. Apollo wrote a SQLite-backed worker for this. This page is a design sketch to be settled when a second application needs one; PhotoTidy, which already has its own event broker, is the likely second consumer. Nothing here is built, and nothing starts until the entry criterion holds.

Context

Apollo does not wait for it; its worker stays in the application. One consumer shows what the code is, not what the shared surface should be, so the sketch records Apollo's shape and the second application revises it.

Async, unlike Apollo's worker. Apollo runs a poller thread and a thread pool over synchronous sessions. The shared worker runs on the event loop over F17.4's AsyncEngine, matching the async package it lives in: a handler is a coroutine, does blocking file or subprocess work through asyncio.to_thread, and reaches a single-owner library through await actor.run(...).

Design

Where. A new rn_forge.sqlalchemy.jobs subpackage behind a jobs extra, over F17.4's engine and migrations. The SSE helper lives in rn_forge.fastapi.sse (F6.2) and is the one piece in the FastAPI package.

Models and migration. Job (id, type string, state, JSON payload, JSON result, error, progress 0–100, progress note, cancel-requested flag, created, started, finished) and JobLog (id, job id with cascade delete, timestamp, level, message), over the package's Base. JobState is pending, running, succeeded, failed, cancelled. type is a string so an application's vocabulary needs no shared enumeration. The package ships an Alembic revision an application includes in its own script location.

JobWorker:

type Handler = Callable[[JobContext], Awaitable[dict[str, Any] | None]]

class JobWorker:
    def __init__(
        self,
        sessions: async_sessionmaker[AsyncSession],
        handlers: Mapping[str, Handler],
        *,
        resources: Mapping[str, object] | None = None,
        on_terminal: Callable[[str], None] | None = None,
        max_workers: int = 2,
        autostart: bool = True,
        history_limit: int = 500,
        idle_poll: float = 2.0,
        shutdown_grace: float = 10.0,
    ) -> None: ...
    async def start(self) -> None: ...
    def notify(self) -> None: ...
    def set_paused(self, paused: bool) -> None: ...
    async def close(self) -> None: ...

start and close run from the application's lifespan. The poller is one asyncio task, and at most max_workers handler tasks run at once.

  1. Bounded claim. The poller claims no more pending jobs than it has idle handler slots, with a conditional UPDATE … WHERE state = 'pending'. Claiming ahead of capacity would show a job as running when it sits in a queue.
  2. Wake and poll. notify() sets an asyncio.Event that wakes the poller on enqueue; the idle poll is the safety net for rows from another process.
  3. Autostart and hold. With autostart=False pending jobs are held, not replayed, until set_paused(False); a count of held jobs is exposed.
  4. Start-up recovery. Jobs left running by the previous process become failed with an "interrupted by restart" error, and history beyond history_limit is pruned oldest first.
  5. Shutdown. close() stops claiming, asks running jobs to cancel, waits up to shutdown_grace, then cancels the remaining handler tasks and logs them.
  6. on_terminal(job_type) runs after every job, success or not.
  7. Terminal recording. A handler's JobCancelled records cancelled; an exception records failed with the traceback logged; asyncio.CancelledError records cancelled and re-raises. A result the JSON column cannot store records failed, never a row stuck at 100% running.

JobContext: await progress(pct, note), await log(level, message) (durable, shown to users), await cancelled() with the database read throttled to once per cancel_poll seconds and sticky once seen, await checkpoint() raising JobCancelled, payload, and resources, the mapping passed to the worker.

The thread actor is an optional resource, not a constructor argument. The worker knows nothing about F17.2. An application that owns an actor passes it as resources={"library": actor}, and a handler reads ctx.resources["library"]. Apollo's worker takes the actor as a required argument because it has one; the shared version must not, or an application without one inherits a concept it does not use. A handler awaits ctx.resources["library"].run(fn).

SSE delta stream. job_events(sessions, *, interval, include_logs) is an async iterator of ServerSentEvents for F6.2's sse_response: a first snapshot, then one frame per job whose state, progress or note changed. Apollo polls once a second, writes its own frame function, sends a comment heartbeat every 30 ticks and wraps a StreamingResponse loop; sse_response and its ping replace all three.

Prior art: Apollo's job worker, its Job and JobLog models and job states, and its job router's delta and stream functions.

Tests (when built): a bounded claim never marks more than max_workers jobs running; a handler's asyncio.to_thread work does not block another job's progress write; a restart fails jobs left running; notify runs a job before the idle poll; cancel is observed at checkpoint within cancel_poll; a failing handler records failed and logs a traceback; an unserialisable result records failed; held jobs do not run until resumed; history pruning keeps the newest history_limit terminal jobs; the delta stream emits a snapshot then only changes.

S17.6.1 — Settle the shared surface with a second application

State: New

Acceptance: a second application's requirements are written into this page, the sketch is revised where they differ from the first, and the open questions are answered or recorded as rejected options.

S17.6.2 — Build the models, worker and context

State: New

Acceptance: rn_forge.sqlalchemy.jobs behaves as under Design with the listed tests passing, an Alembic revision applies to an empty SQLite database through F17.4's run_migrations, and the worker imports nothing from rn_forge.commons.runtime.actor.

S17.6.3 — Build the SSE delta stream

State: New

Acceptance: job_events feeds sse_response in a test application; a client reads a snapshot and then only the changed job.

Acceptance

Written when S17.6.1 settles the design; each line is tagged with its story.

Considered and rejected

  • A broker-backed queue (Celery, a message bus). Local-first applications cannot ask the user to install and supervise a second service.
  • The actor as a constructor argument. See Design.
  • A thread-pool worker over synchronous sessions, as Apollo has. The package is async only (F17.4), and an async worker leaves only the single-owner library on a dedicated thread.

Open questions

  • Should the handler registry key on a string, as sketched, or on an application-supplied enumeration?
  • Does PhotoTidy's event broker want the same persisted log, or only progress?