Knowledge pipeline
How a document becomes something an agent can find and cite — added, read, indexed and searched, with the cost of each step.
A knowledge baseKnowledge baseKho tri thứcA collection of documents agents can search and cite. Files are uploaded or synced from SharePoint, and each knowledge base has its own sharing rules.Open in glossary document is two things: its bytes in the S3 object store, and rows in the retrieval store — pgvector inside the portal’s own Postgres. That store is shared with memory: a document passage and a remembered statement are rows of one table, read through the knowledge base both are filed under (internally, a “collection”). Extraction, chunking and embedding run inside the API process.
Key rules
- Documents are read once, when they are added or change — not on every question.
- Sharing rules reach every search, including an agent’s. See resource visibility — it covers a knowledge base and everything filed under it.
- Reading is billed to the uploader’s department; searching to the searcher’s.
- Keyword search is always on and free.
- A document’s text is searchable while it is still Pending.
Upload a file, or add a SharePoint folder that syncs on a schedule. Each file is stored once, keyed by its content, so the same bytes attached again match the copy already there.
Upload request
An upload answers 202: the document row exists, but nothing is searchable yet. The request:
-
Checks the knowledge base
Confirms the knowledge base exists in your organisation and has a manual source.
-
Stores the bytes
The file goes to the object store.
-
Creates or supersedes the document
A new document is created, or the previous one with the same uri is superseded. The form may carry a reading levelReading levelMức đọcHow deeply the portal reads a PDF or image: Text (text layer and OCR, no model), Text + figures (pictures described), or Full rewrite (a model writes out every page). Deeper costs more.Open in glossary and a semantic search switch for this file; a field left out follows the manual source.
-
Queues one job
One ingest job per document.
Registering a document by reference is a separate route that writes no bytes. A file the portal cannot read is refused at upload, while the dialog is still open — see file formats.
Chat attachments
An attachment is a document or a picture, decided once.
| Kind | Handling |
|---|---|
| Document | Indexed when uploaded, and described in the message by a manifest the model reads it through. SVG is refused, as a document that can execute. |
| Picture | Handed to the model as itself; no words are pulled from a photograph and no spreadsheet is sent as an image. Formats and the size ceiling are the provider’s, enforced at upload. A picture is stored without a document. |
Where documents go. A document is uploaded when it is attached, before the message is sent. Each person has one attachments knowledge base, created on their first attachment, owned by them, carrying no rules and hidden from every listing. It holds one manual source read at Text, with semantic search on while the organisation has an embedding model assigned and off while it has none. Ingest is the same job as any upload — text layer first, OCR for pages that need it, no model call — billed to the uploader’s department.
Same bytes, same document. The document’s uri is the file’s SHA-256, not its name. Two files sharing a name stay apart; the same bytes attached again, in any conversation, match the live document and queue nothing. A failed or stopped one is superseded and queued again.
What a turn may read. Only the documents attached in its own conversation, recorded on each message attachment. The rest of the person’s attachments are out of reach of that conversation.
The manifest. The message carries one entry per document, inside the same quoted-material framing as the attachment: filename and type, an attachment id, how far reading has got, which pages could not be read, and the first 20 section headings.
- A finished document of at most 6,000 characters is placed in the message whole, because a tool call costs more than that much text.
- Anything longer is read with
read_document(reaches any attachment of this conversation whatever the agent binds, and can read a range of pages) and searched withsearch_attachments(reaches only this conversation’s attachments). - A read of a document still being indexed waits up to a minute, then answers with what is there and the pages still missing.
Pictures. A picture larger than the provider looks at is reduced before it is sent, and the message names the size it was reduced to — the provider caps an image at a longest edge and a number of visual tokens and silently rescales anything above either. A twelve-megapixel photo sent whole would cost bytes on every later turn and arrive no sharper. The reduction runs where the bytes already are; a picture within both caps passes byte for byte. A message with a picture needs a model that can see: the turn is refused before anything is recorded, naming the model. The composer warns before send, but the refusal is the authority, since a per-turn override never passes through the composer.
Later turns. Every earlier document replays as its manifest, built from the index with its current status, so no file is downloaded or read again. Inline text replays within a shared budget, newest first; a document past the budget replays without it, to be read again by tool. The few most recent picture-bearing messages resend their pictures (the count is configuration, because every past turn is paid for on every request). Older messages say in words that their picture is gone and name it; a picture retention or storage cannot return is left out. Picture release inside a run is covered in the agentic loop.
Indexing settings
Every file is indexed with two settings: a reading level and a semantic search switch. A source holds the default for its files — Text with semantic search on, unless whoever created it chose otherwise. A file may hold its own value for either half; a half it leaves empty follows the source. Both are resolved once, when the job is queued, and travel on the job.
Word, Excel and other typed formats are always read as Text. Going back to a shallower level keeps what was already read and costs nothing.
| Reading level | What it takes out of a file | Model calls |
|---|---|---|
Text (TEXT) | The text layer, plus OCR for every page the text layer cannot serve | None |
Text + figures (FIGURES) | The same text, plus a description of the pictures on every page that paints one | One per page with a picture |
Full rewrite (REWRITE) | Every page written out as Markdown by the background model, checked against its text layer or OCR, pictures described where they sit | One per page |
- Keyword only costs nothing and keyword indexing is always on. Semantic search is the embedding, one call per batch of passages. A file with semantic off is chunked and indexed for keywords only — which is also how files index in an organisation with no embedding model assigned.
- Text + figures and Full rewrite reach only a PDF or an image. A typed format has no pages, so it is read at Text whatever it asks for.
- A page “paints a picture” when it draws a raster image at least 150 pixels on its shorter side. A scanned page is one image, so under Text + figures every scanned page costs a call; the model answers nothing for a page showing no picture. A chart drawn as vectors paints no image and is not described.
- Full rewrite is for reading the file beside its transcript, where Markdown headings and tables make a misread word easier to spot. Retrieval reads the same words under Text.
Changing a setting queues a file only when the change deepens it: a deeper level, or semantic search switched on. Going shallower queues nothing and keeps the passages already written. Changing a source’s default queues every file that follows the source and is deepened by it; a file holding its own value for the half that changed is left alone. A file whose stored bytes are gone is counted and skipped, and the next sync fetches it again.
What one ingest does
| Step | What it adds | Cost |
|---|---|---|
| Text layer | Chunked, embedded when semantic search is on, and indexed — the whole job for a typed document | About one embedding call |
| Pages the text layer cannot serve | Read by OCR; each transcript filed where that page’s section would go | Nothing |
| Pictures, under Text + figures | Described by the background model and filed on their page | One model call per page with a picture |
| Every page, under Full rewrite | Written out by the background model; transcripts replace the text layer’s passages | One model call per page |
A document is PENDING while any of that is still happening and READY when all of it has finished, unread pages counted included. Its text passages are already searchable while it reads PENDING — the status describes the run, not the index.
Every passage is embedded behind its document’s title, which costs nothing and gives a lone paragraph the context it would otherwise miss. Its own content already carries the section path as a heading. How text is cut into passages is on chunking and sections.
Pages and merge
One page failing costs one page.
- A page OCR read nothing from is counted unread.
- Under Full rewrite, a page the model answers with nothing — or does not answer within the time allowed — is read again on the image already rendered, up to the attempt ceiling. It then keeps what OCR or the text layer gave it, or is counted unread if there is neither.
- A picture the model does not answer in time is asked again, up to the same ceiling.
The merge runs once, when every page the job asked for has settled, and covers only the pages this run read. Under Text and Text + figures it replaces what OCR and a model read, leaves the text layer’s passages alone, and keeps picture descriptions an earlier figures run paid for. Under Full rewrite every page has its own transcript, so the merge replaces the text layer’s passages too; until it does, the text layer is searchable.
Transcripts and descriptions are stored rows, so a job that runs again — a retry, a re-process or a deeper setting — pays only for what is outstanding. A page OCR wrote is read again under Full rewrite; a page a model wrote is never paid for twice, and stays in the index when the file is re-processed at Text.
Job queue
One job covers one document, on the shared queue described in background jobs. Each backend instance registers a handler with a lane count from configuration; zero lanes means no handler (a request-only replica or a CLI process). Every model call has a deadline the gateway clients arm from configuration, and no caller may waive it.
Inside a job, pages are rendered in as few subprocess calls as there are runs of consecutive pages, then read under a second limit on how many pages any instance has in flight. That lets a hundred-page document drain through the lanes rather than hold them all.
| Event | Result |
|---|---|
| A page comes back unreadable from the model | Tried again on the image it already has, then keeps its OCR or text-layer reading, or is counted unread |
| The model does not answer in time | Treated as an empty answer: OCR or the text layer stands in, or the page is counted unread |
| Model call outlives its deadline | A failed attempt, reported in a sentence |
| Attempt fails, attempts remain | Back to the queue with doubling backoff |
| Attempts run out | Job goes to a dead letter queue, the document is failed, and the reason is written where a person reads it |
| Worker dies mid-ingest | The attempt expires and the queue hands the job out again; stored transcripts mean no page is paid for twice |
Stalled runs. A whole run can die — a worker stops, a deploy replaces it, or the gateway refuses until attempts are spent — leaving the document pending with nothing working on it. A sweep every five minutes finds pending rows that have gone quiet longer than the configured window (a run writes to the row as each page lands) and puts them back on the queue, billed to the department recorded on the document at queue time. Each restart is counted; past the configured ceiling the document fails with a sentence saying to re-process it. A document with no recorded department (nobody to bill) or whose stored copy is gone (nothing to read) is failed the same way rather than restarted. Asking for a re-process resets the count.
Retirement and failure
A re-ingest retires the previous revision’s passages before inserting the new ones, so a failure between the two leaves nothing answering rather than a withdrawn procedure. Retirement is the mark recall already filters on; the passage count and date update when it commits.
Only the text layer can fail a document; a page that exhausts its attempts is counted unread. The document is marked failed only when the job’s attempts run out, so it does not flicker between pending and failed during retries. The reason is a whole sentence shown verbatim.
| Failure | Reason shown |
|---|---|
| Unsupported format | Names the formats that work |
| Scan with no text layer | Says so |
Stops that end a model reading early are listed on file formats.
Re-processing and stopping
| Control | Effect |
|---|---|
| Re-process | Runs the pipeline again over bytes the portal holds, at the file’s current settings. The row goes back to pending before the job is queued; a pending row whose queueing failed is settled by pressing again. |
| Indexing | Sets the file’s own reading level and semantic search, or hands either back to the source; queues the file when the change deepens it. |
| Stop | Settles both rows at once — job cancelled, document saying so — and lets the attempt give up on its own. The run checks for cancellation before pages are read, before passages are embedded and before anything reaches the knowledge base; anything produced after that is discarded. |
A stopped document has a state of its own, with its recorded reason cleared, and whatever the last finished run indexed still answers. These controls answer 202, so the pressed button stays in flight until the document list is re-read, and a run under way shows Stop beside it. For a SharePoint source, Pause stops a sync pass — see SharePoint sync.
Who pays
Every model call an ingest makes — rewriting a page, describing a page’s pictures, embedding a passage — runs on the uploader’s department’s virtual keyVirtual keyKhoá ảoA gateway key issued per department, so each model call is limited and billed to the right department. People never handle it.Open in glossary, so that department’s budget and blocked models bound the upload. The department is the uploader’s at upload time, recorded on the job. Each call leaves the same audit row a chat turn leaves, naming no agent, with Document ingest where an agent’s name would be.
Retrieval is billed the same way: embedding a query runs on the searcher’s department key, so the department’s budget bounds a search. The deployment’s admin key provisions the gateway and issues keys, and calls no model.
| Call | Reported |
|---|---|
| Embedding | Prompt tokens and route |
| Page rewrite, picture description | Full token partition, including what the cache served |
Spend is priced at the rates the gateway reported for that route and written twice: onto the document, by the three kinds of call, and into the ingest ledger, whose line names the paying department. A count the gateway did not report, or a route nobody priced, leaves the cost unknown — zero is never written. See audit and cost.
Rows in the store
A passage holds three texts, derived together, so it cannot be reworded in place: retiring it is offered, and re-ingesting changes its wording.
| Text | Content |
|---|---|
| Embedded | The document title, then the passage with its headings |
| Shown | The passage alone |
| Keyword and literal indexes | The document title and the body without headings — a repeated heading would rank every chunk of a section alike |
Why it works this way
Why read documents up front instead of at question time?
Reading — especially scans and model reading — is slow and costs money. Doing it once makes every later question fast and cheap, and lets you check the reading before anyone relies on it.
Why is a document searchable while it still says Pending?
Pending describes the reading run, not the index. Text passages are searchable as soon as they are stored while the deeper steps continue; when everything finishes the document turns Ready and the merge replaces the first passages where needed.
Why retire passages before inserting new ones?
If a re-ingest fails halfway, it is safer that nothing answers than that a withdrawn procedure keeps being cited.
Under the hood
Storage details
Every passage insert is raw SQL, because the embedding column is a vector type the ORM does not model; the repository has no create method. Ingest spend lands in usage.ingest_costs. Each ingest’s working time and reading level are written to retrieval.ingest_timings (one row per document, last run) and feed the SharePoint first-sync estimate.