Skip to content

Worker & Progress

Summary

Long-running work runs in a background worker. Jobs are durable documents that move through an explicit status machine. Progress is streamed to clients.

The status machine

1
2
3
pending -> processing -> completed
                     \-> failed    (retried up to N times)
                     \-> paused    (waiting for human input)
  • pending — queued, eligible for pickup.
  • processing — claimed by a worker.
  • paused — blocked on a human-in-the-loop question.
  • failed — error; retry_count increments until a max, then skipped.
  • completed — terminal success.

The poll loop

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
async def poll(resources):
    backoff = MIN_POLL
    while True:
        jobs = await job_svc.search(filter={"status": "pending"}, limit=100)
        backoff = MIN_POLL if jobs else min(backoff * 2, MAX_POLL)
        for job in jobs:
            if (job.retry_count or 0) >= MAX_RETRIES:
                continue
            await job_svc.update_status(job.id, "processing")
            try:
                await process(job, resources)
            except PauseForResolution:
                await job_svc.update_status(job.id, "paused")
            except Exception as e:
                await job_svc.update_status(
                    job.id, "failed", error=str(e),
                    retry_count=(job.retry_count or 0) + 1,
                )
        await asyncio.sleep(backoff)

Tunables live in environment variables: WORKER_POLL_MIN, WORKER_POLL_MAX, WORKER_MAX_RETRIES.

Progress events

A worker publishes structured progress events to a bus (a port). The HTTP edge exposes a streaming endpoint that replays events for a given job id.

1
2
3
4
5
class ProgressEvent(BaseModel):
    job_id: str
    step: str
    status: str            # started | progress | completed | skipped | paused
    message: dict | None = None

The bus is an in-memory adapter by default; because it is a port, it can be swapped for a durable broker. Emit from the orchestrator at each stage:

1
2
await self.bus.publish(ProgressEvent(job_id=jid, step="normalize",
                                     status="completed", message={"count": n}))

Streaming endpoint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
async def stream_job_events(request, id):
    bus = request.app.state.db_resources.get("progress_bus")
    if bus is None:
        return {"events": []}

    async def gen():
        sent = 0
        while True:
            events = bus.get_events(id)
            for e in events[sent:]:
                yield f"data: {e.model_dump_json(exclude_none=True)}\n\n"
                sent += 1
            await asyncio.sleep(0.5)

    return StreamingResponse(gen(), media_type="text/event-stream")

Known limitation. The default in-memory bus is per-process. If the worker and the API run as separate processes, events published by the worker are not visible to the API. For production, back the bus with a shared broker. The port boundary makes this a one-adapter change.