Background jobs
The work queue behind ingest, SharePoint sync and gateway sync — which queues exist, where their tables live, and how failed attempts are retried.
Work the portal defers, rather than doing inside a request, runs on a queue in the portal’s own Postgres, through the pg-boss library. There is no separate queue service to deploy.
Key rules
- The queue lives in Postgres. Its tables sit in their own
pgbossschema. - Exclusive queues run one job per key. Two passes never work on the same folder or department.
- A failed attempt is retried after a doubling delay, up to a ceiling the deployment sets.
- A job that runs out of attempts lands in a dead letter queue, and the item it worked on is marked failed where a person can see it.
Queues
| Queue | Carries | Policy |
|---|---|---|
knowledge.ingest | One document to extract, read, chunk and index — see Knowledge pipeline | Standard, so many documents run at once |
knowledge.ingest.sweep | Nothing; a cron fires it to queue pending documents nothing has written to for a while | Exclusive |
knowledge.sync | One SharePoint source to pull — see SharePoint sync | Exclusive per source |
knowledge.sync.sweep | Nothing; a cron fires it | Exclusive, so a slow pass never stacks |
agent.turn.sweep | Nothing; a cron fires it to end turns still running past their ceiling — see Turn lifecycle | Exclusive |
gateway.department.sync | One department whose LiteLLM team to reconcile | Exclusive per department |
gateway.department.sync.sweep | Nothing; a cron fires it to queue every active department | Exclusive |
An exclusive queue admits one job per singleton key, whether queued or active. That is what keeps two passes off one folder. A standard queue runs as many jobs at once as its workers allow.
A job’s life
- Queued — a request, a cron or another job adds it.
- Active — a worker picks it up.
- Retry — a handler that throws fails one attempt. The queue tries again after a delay that doubles each time, up to the attempt ceiling the deployment configures.
- Dead letter — a job whose attempts run out lands in a dead letter queue, and whatever it was working on is marked failed where a person reads it.
An attempt that outlives its expiry is taken back and retried. That is how a worker that dies mid-job is recovered, without a sweep of its own.
Where the tables live
The library owns every table in a schema of its own, pgboss, which the deployment creates — see the readiness checklist.
- Building and upgrading those tables needs data-definition rights, so it happens during the schema apply, under the migrating role.
- The API starts the library with migration switched off. The API’s database role holds no data-definition rights anywhere, and this separation preserves that.
- Nothing in the portal’s own schema list names
pgboss, so pushing the portal’s declared schema neither builds nor drops any of it.
Workers
- Each process registers one handler per queue at boot, with a lane count from configuration.
- Dispatch travels on Postgres notifications rather than a polling interval.
- A deployment that configures no lanes registers no handler. That is how a request-only replica stays one.
Why it works this way
Why exclusive queues for sweeps and syncs?
A sweep fired by a cron can take longer than the interval between firings. Admitting one job per key means a slow pass never stacks behind itself, and two passes never pull the same SharePoint folder or reconcile the same department at once.
Under the hood
How modules use the queue
A module that queues work depends on a port, never on pg-boss itself. The schema is applied by apply-schema under the migrating role; in development the init SQL creates the pgboss schema, and in AWS the account’s own infrastructure does.